Databricks Architect (Lead Data Platform Architect)
New York, United States
USD 180K-247K (estimate) Senior-level Full Time
Tasks
- Architect batch and real time processing
- Architect data governance frameworks
- Architect structured and unstructured workloads
- Build data ingestion frameworks
- Collaborate on networking security and storage strategies
- Conduct design reviews
- Define Delta Lake data storage standards
- Define job orchestration standards
- Define workspace design and cluster strategy
- Design ETL and ELT pipelines
- Design data modeling standards
- Design lakehouse architecture end to end
- Enforce architecture governance
- Ensure data privacy compliance
- Guide Spark and PySpark optimization best practices
- Implement Databricks SQL
- Implement MLflow
- Implement Spot instances
- Implement Unity Catalog
- Implement autoscaling
- Implement cost governance strategies
- Implement data lineage and cataloging
- Implement fine grained access control
- Implement job clusters
- Implement medallion architecture
- Implement metadata management
- Integrate Databricks with BI tools
- Integrate Databricks with ML pipelines
- Integrate Databricks with data sources
- Lead Databricks platform deployment and optimization
- Mentor engineering teams
- Optimize cluster utilization
- Optimize query performance
- Optimize storage efficiency
Perks/Benefits
- N/A
Skills/Tech-stack
ABAC | AWS | Airflow | Apache Spark | Autoscaling | Azure | Azure Data | Azure Data Factory | Azure Event | Azure Event Hubs | Cost Optimization | Data Factory | Data Governance | Data Ingestion | Data Lineage | Data Modeling | Data Vault | Data mesh | DataOps | Databricks | Databricks SQL | Delta Lake | Dimensional Modeling | ELT | ETL | Event Hubs | GCP | GenAI | Job Clusters | Kafka | MLOps | MLflow | Metadata Management | Photon | PySpark | RBAC | SQL | Spot instances | Star Schema | Unity Catalog
Education
N/A
Related jobs
-
Sr. Data Engineer USD 98K-130KAgile | Airflow | CI/CD | DBT | Data ArchitectureDental insurance | Life insurance | Medical insurance | Paid Holidays | Paid time offSenior-level Full TimeLatham, NY, United States3h ago
-
AWS S3 | Airbyte | Apache Airflow | Apache Hudi | Apache Iceberg401k plan | Disability insurance | Employee assistance program | Employee discount program | Health/Dental/VisionSenior-level Full TimeRemote - Nationwide, United States R4h ago
-
Data Engineer USD 99K-130KBusiness Objects | Change Management | Data Governance | Data Lineage | Data ModelingMid-level Full TimeNashville, TN, United States6h ago
-
Data Engineer USD 101K-132KAgile | Amazon Web Services | Data Engineering | Data Modeling | Data PipelinesMid-level Full TimeAuburn Hills, MI, United States6h ago
-
Mid-level Full TimeCenter, Center District, IL6h ago
-
Senior-level Full TimeUS-OH-Columbus8h ago
-
Senior Data Engineer USD 130K-226KAzure | Azure Machine Learning | Azure Synapse | CI | CI/CD401k savings plan | Dental insurance | Employee recognition | Employee referral bonuses | HealthcareSenior-level Full TimeWashington, District of Columbia, United States9h ago
-
Data Engineer, Data Architecture and Engineering, gData USD 130K-188KC++ | Data Modeling | Data Processing | Data Warehouse | Distributed dataSenior-level Full TimeSunnyvale, CA, USA; Boulder, CO, USA10h ago
-
Senior Solution Architect – Databricks USD 123K-176KAmazon Web Services | Apache Spark | Asset bundles | Azure | CI/CD401k savings plan | Dental insurance | Flexible Spending Accounts FSAs | Health Savings Account (HSA) | Health insuranceSenior-level Full TimeUS-Nationwide-FIELD, United States21h ago
-
AWS | Azure | CI/CD | Data Modeling | Data QualitySenior-level Full TimeTwo Destiny Way, Westlake TX, United … R21h ago
-
Senior Data Engineer USD 91K-163KAKS | Alerting | Azure Data | Azure Data Factory | Azure Databricks401k contribution | Career development opportunities | Comprehensive benefits package | Equity stock purchase | Incentive and recognition programsSenior-level Full TimePrimary location: Eden Prairie, MN R21h ago
-
Data Engineer, Senior USD 77K-176KAWS | Agile | Amazon Redshift | Apache Kafka | Apache SparkDependent care | Health insurance | Paid leave | Professional development | Retirement benefitsSenior-level Full TimeUSA, VA, McLean (8283 Greensboro Dr, …21h ago
-
Senior-level Full TimeOne Destiny Way, Westlake TX, United … R21h ago
-
AI | AWS | Analytics | Autonomous Database | Azure401k match | Dental insurance | Health insurance | Life insurance | Paid HolidaysSenior-level Full TimeUnited States21h ago
-
Data Engineer USD 91K-146KAPI Integration | Data Governance | Data Migration | Data Validation | Data integration401k plan | Dental insurance | Long-term disability insurance | Medical insurance | Paid leaveMid-level Full TimeHanover, MD23h ago
-
APIs | AWS Glue | AWS Redshift | Amazon Web Services | Apache AirflowFully remote | Health insurance | On call production support | Paid time off | Retirement plansSenior-level Full TimeOrlando, FL, United States R1d ago
-
Analytics Engineer USD 132K-173KAgile | Apache Airflow | Automated testing | CI/CD | DBTHigh pressure fast moving environment | In person work 3 days per week | User testing and QA involvementMid-level Full TimeUS-CA-Menlo Park1d ago
-
Senior-level Full TimeDes Moines, IA, United States1d ago
-
Staff Data Engineer USD 24K-24KAWS | Apache Airflow | Apache Spark | Argo Workflows | Containerization401k match | Flexible time off | Health insurance | Health savings account | MacBook providedSenior-level Full TimeBeavercreek, OH1d ago
-
Binary Data Formats | Binary data | C++ | Camera Calibration | Cloud processingMid-level Full TimeMountain View, CA1d ago
-
Senior Software Engineer, Data Platform USD 190K-250KApache Flink | Apache Kafka | Apache Spark | Batch Processing | Change Data Capture401k | Coworking stipend | Dental insurance | Employer pension contribution | Family planning benefitsSenior-level Full TimeSan Francisco1d ago
-
Lead Data Engineer - Refer USD 316K-506KContinuous Delivery | Data Architecture | Data Engineering | Data Governance | Data LakeLearning and development programs | Remote workSenior-level Full TimeChicago, Illinois, USA R1d ago
-
Lead Data Engineer USD 316K-506KContinuous Delivery | Data Architecture | Data Engineering | Data Governance | Data LakesLearning and development programs | Mentoring and coaching | Remote workSenior-level Full TimeChicago, Illinois, USA R1d ago
-
Cloud Data Engineer USD 119K-208KAzure | Azure Data | Azure Data Factory | Azure SQL | Azure SynapseMid-level Full TimeIndianapolis, IN, US1d ago
-
Staff, Analytics Engineer, GTM Data Science & Analytics USD 155K-228KAutomated reconciliation | CI/CD | DBT | Data Modeling | Data ObservabilitySenior-level Full TimeRemote - US R1d ago