Databricks Architect (Lead Data Platform Architect)
New York, United States
USD 180K-247K (estimate) Senior-level Full Time
Tasks
- Architect batch and real time processing
- Architect data governance frameworks
- Architect structured and unstructured workloads
- Build data ingestion frameworks
- Collaborate on networking security and storage strategies
- Conduct design reviews
- Define Delta Lake data storage standards
- Define job orchestration standards
- Define workspace design and cluster strategy
- Design ETL and ELT pipelines
- Design data modeling standards
- Design lakehouse architecture end to end
- Enforce architecture governance
- Ensure data privacy compliance
- Guide Spark and PySpark optimization best practices
- Implement Databricks SQL
- Implement MLflow
- Implement Spot instances
- Implement Unity Catalog
- Implement autoscaling
- Implement cost governance strategies
- Implement data lineage and cataloging
- Implement fine grained access control
- Implement job clusters
- Implement medallion architecture
- Implement metadata management
- Integrate Databricks with BI tools
- Integrate Databricks with ML pipelines
- Integrate Databricks with data sources
- Lead Databricks platform deployment and optimization
- Mentor engineering teams
- Optimize cluster utilization
- Optimize query performance
- Optimize storage efficiency
Perks/Benefits
- N/A
Skills/Tech-stack
ABAC | AWS | Airflow | Apache Spark | Autoscaling | Azure | Azure Data | Azure Data Factory | Azure Event | Azure Event Hubs | Cost Optimization | Data Factory | Data Governance | Data Ingestion | Data Lineage | Data Modeling | Data Vault | Data mesh | DataOps | Databricks | Databricks SQL | Delta Lake | Dimensional Modeling | ELT | ETL | Event Hubs | GCP | GenAI | Job Clusters | Kafka | MLOps | MLflow | Metadata Management | Photon | PySpark | RBAC | SQL | Spot instances | Star Schema | Unity Catalog
Education
N/A
Related jobs
-
CDN | Cloud infrastructure | Dashboards | Data Modeling | Data PipelinesEntry-level Full TimeSeattle, Washington, United States3h ago
-
Senior Data Engineer (Multiple Positions) USD 220K-387KApache Flink | Apache Spark | Architecture Design | Data Governance | Data ModelingSenior-level Full TimeSan Jose, California, United States3h ago
-
Staff Data Engineer USD 145K-180KAgile | Algorithms | Automation | Cloud Architecture | Data ComplianceCompany 401k match | Family-forming benefits | Financial Planning Sessions | Flexible time off | Free snacks and refreshmentsSenior-level Full TimeAustin, Texas, United States12h ago
-
Consultant, Cybersecurity (Data Engineer) USD 118K-222KAccess Control | Apache Kafka | Apache Spark | Application Security | Automation401k matching | Health and wellbeing benefits | Hybrid schedule | Paid time off | Remote internal associates consideredSenior-level Full TimeOhio - Columbus, Three Nationwide Plaza, …15h ago
-
Human Data Manager USD 100K-186KAI Agents | Alignment Evaluation | Data Annotation | Data Generation | Data Integrity401k | Dental insurance | Disability insurance | Life insurance | Medical insuranceMid-level Full TimePalo Alto, California16h ago
-
Senior Data Engineer (ETL / Python Developer) USD 120K-163KApache Spark | Azure | Azure Data | Azure Data Factory | Azure DevOpsSenior-level Full TimeIndianapolis, IN, US16h ago
-
Finance & Strategy AI Engineer USD 146K-246KAPI Integration | Adaptive Planning | Agentic Workflows | BigQuery | Data GovernanceFlexible remote work | Health benefits | Parental leave | Professional development stipendSenior-level Full TimeSan Francisco - SF9 R17h ago
-
Senior-level Full TimeCupertino19h ago
-
Lead Data Engineer - Auctions & Outcomes USD 200K-230KAWS | Airflow | Apache Iceberg | Apache Kafka | Apache SparkSenior-level Full TimeNew York, New York19h ago
-
Senior Software Engineer, Data Platform USD 196K-245KAirflow | BigQuery | Cost Optimization | Dagster | Data GovernanceBenefits | Equity | Relocation assistanceSenior-level Full TimeSan Francisco Bay Area or Remote … R19h ago
-
Senior Data Architect USD 134K-205KAWS | Apache Spark | Azure | Cloud Native | Cloud-native DatabasesComprehensive benefits program | Performance-Based IncentivesSenior-level Full TimeHuntsville, AL20h ago
-
Data Engineer - DataOps USD 130K-150KApache Airflow | Apache Iceberg | Apache Spark | CI/CD | Cause analysis401k match | Career growth | Hybrid work schedule | Medical, dental, and vision insurance | Paid parental leaveMid-level Full TimeCupertino, California, United States20h ago
-
Senior Data Engineer - AI Platform USD 200K-230KClickHouse | Data Pipeline Reliability | Data pipeline | Debugging | Distributed SystemsSenior-level Full TimeUnited States R20h ago
-
Mid-level Full TimePhoenix, AZ, United States20h ago
-
Data Engineer, Knowledge Graph USD 200K-300KAmazon Neptune | Data Modeling | Data Transformation | Data integration | Graph Algorithms401k savings program | Family leave | Health care benefits | Mental wellness programs | Parental leaveSenior-level Full TimeNew York, NY20h ago
-
Senior-level Full TimeMemphis, Tennessee, United States20h ago
-
Sr Manager, Data Platform & Decision Support USD 175K-195KAgile | Automation | BI | Business Intelligence | CI/CDSenior-level Full TimeMinneapolis, MN, United States20h ago
-
Senior AI Solutions Architect USD 145K-200KAI Engineering | Automation | Cloud Computing | Data integration | Data platforms401k match | Employee discounts | Health & wellness programs | PTO | Relocation assistanceSenior-level Full TimeFramingham, MA, United States20h ago
-
Teamcenter Data Architect USD 113K-154KActive Workspace | BMIDE | Configuration Management | Confluence | Data Governance401k matching | Employee assistance program | HSA and FSA options | Life insurance | Medical, dental, and vision insuranceSenior-level Full TimeRemote (United States) R21h ago
-
Specialist Solutions Architect - Data Engineering & Warehousing (Digital Native Business) USD 180K-247KAWS | Anomaly Detection | Apache Spark | Cloud platform | Data ModelingContinuous learning | Mentorship | Remote work | Travel up to 30 percentSenior-level Full TimeUnited States R21h ago
-
Senior AI Data Engineer USD 118K-178KAI Agent | BigQuery | CRM | Customer Data | Customer Data PlatformFlexible working model | Health insurance | Parental leave | Professional development stipend | Remote workSenior-level Full TimeRemote - US R22h ago
-
Senior-level Full TimeRedwood City, California, United States22h ago
-
Argo CD | Big Data | Concurrency | Distributed Systems | Fine TuningEntry-level Full TimeAustin R22h ago
-
Lead Software Engineer - Data Engg/ Data Analytics USD 170K-186KAWS | Agile | Airflow | Apache Kafka | Apache SparkSenior-level Full TimeHouston, TX, United States22h ago
-
Data Engineer I - Salesforce Data 360 USD 87K-123KAPIs | Access Controls | Data Governance | Data Lineage | Data Modeling401k match | Career growth and development programs | Comprehensive onboarding | Conference participation | Employee resource groupsMid-level Full TimeMorrisville, North Carolina (NC), United States R22h ago