Databricks Architect (Lead Data Platform Architect)
New York, United States
USD 180K-247K (estimate) Senior-level Full Time
Tasks
- Architect batch and real time processing
- Architect data governance frameworks
- Architect structured and unstructured workloads
- Build data ingestion frameworks
- Collaborate on networking security and storage strategies
- Conduct design reviews
- Define Delta Lake data storage standards
- Define job orchestration standards
- Define workspace design and cluster strategy
- Design ETL and ELT pipelines
- Design data modeling standards
- Design lakehouse architecture end to end
- Enforce architecture governance
- Ensure data privacy compliance
- Guide Spark and PySpark optimization best practices
- Implement Databricks SQL
- Implement MLflow
- Implement Spot instances
- Implement Unity Catalog
- Implement autoscaling
- Implement cost governance strategies
- Implement data lineage and cataloging
- Implement fine grained access control
- Implement job clusters
- Implement medallion architecture
- Implement metadata management
- Integrate Databricks with BI tools
- Integrate Databricks with ML pipelines
- Integrate Databricks with data sources
- Lead Databricks platform deployment and optimization
- Mentor engineering teams
- Optimize cluster utilization
- Optimize query performance
- Optimize storage efficiency
Perks/Benefits
- N/A
Skills/Tech-stack
ABAC | AWS | Airflow | Apache Spark | Autoscaling | Azure | Azure Data | Azure Data Factory | Azure Event | Azure Event Hubs | Cost Optimization | Data Factory | Data Governance | Data Ingestion | Data Lineage | Data Modeling | Data Vault | Data mesh | DataOps | Databricks | Databricks SQL | Delta Lake | Dimensional Modeling | ELT | ETL | Event Hubs | GCP | GenAI | Job Clusters | Kafka | MLOps | MLflow | Metadata Management | Photon | PySpark | RBAC | SQL | Spot instances | Star Schema | Unity Catalog
Education
N/A
Related jobs
-
Sr. Data Engineer USD 98K-130KAgile | Airflow | CI/CD | DBT | Data ArchitectureDental insurance | Life insurance | Medical insurance | Paid Holidays | Paid time offSenior-level Full TimeLatham, NY, United States2h ago
-
AWS S3 | Airbyte | Apache Airflow | Apache Hudi | Apache Iceberg401k plan | Disability insurance | Employee assistance program | Employee discount program | Health/Dental/VisionSenior-level Full TimeRemote - Nationwide, United States R3h ago
-
Data Engineer USD 99K-130KBusiness Objects | Change Management | Data Governance | Data Lineage | Data ModelingMid-level Full TimeNashville, TN, United States4h ago
-
Mid-level Full TimeCenter, Center District, IL5h ago
-
Senior-level Full TimeUS-OH-Columbus7h ago
-
Senior Data Engineer USD 130K-226KAzure | Azure Machine Learning | Azure Synapse | CI | CI/CD401k savings plan | Dental insurance | Employee recognition | Employee referral bonuses | HealthcareSenior-level Full TimeWashington, District of Columbia, United States8h ago
-
Data Engineer, Data Architecture and Engineering, gData USD 130K-188KC++ | Data Modeling | Data Processing | Data Warehouse | Distributed dataSenior-level Full TimeSunnyvale, CA, USA; Boulder, CO, USA8h ago
-
Senior Solution Architect – Databricks USD 123K-176KAmazon Web Services | Apache Spark | Asset bundles | Azure | CI/CD401k savings plan | Dental insurance | Flexible Spending Accounts FSAs | Health Savings Account (HSA) | Health insuranceSenior-level Full TimeUS-Nationwide-FIELD, United States20h ago
-
AI | AWS | Analytics | Autonomous Database | Azure401k match | Dental insurance | Health insurance | Life insurance | Paid HolidaysSenior-level Full TimeUnited States20h ago
-
Data Engineer USD 91K-146KAPI Integration | Data Governance | Data Migration | Data Validation | Data integration401k plan | Dental insurance | Long-term disability insurance | Medical insurance | Paid leaveMid-level Full TimeHanover, MD22h ago
-
APIs | AWS Glue | AWS Redshift | Amazon Web Services | Apache AirflowFully remote | Health insurance | On call production support | Paid time off | Retirement plansSenior-level Full TimeOrlando, FL, United States R22h ago
-
Analytics Engineer USD 132K-173KAgile | Apache Airflow | Automated testing | CI/CD | DBTHigh pressure fast moving environment | In person work 3 days per week | User testing and QA involvementMid-level Full TimeUS-CA-Menlo Park22h ago
-
Senior-level Full TimeDes Moines, IA, United States22h ago
-
Staff Data Engineer USD 24K-24KAWS | Apache Airflow | Apache Spark | Argo Workflows | Containerization401k match | Flexible time off | Health insurance | Health savings account | MacBook providedSenior-level Full TimeBeavercreek, OH22h ago
-
Binary Data Formats | Binary data | C++ | Camera Calibration | Cloud processingMid-level Full TimeMountain View, CA22h ago
-
Senior Software Engineer, Data Platform USD 190K-250KApache Flink | Apache Kafka | Apache Spark | Batch Processing | Change Data Capture401k | Coworking stipend | Dental insurance | Employer pension contribution | Family planning benefitsSenior-level Full TimeSan Francisco23h ago
-
Lead Data Engineer - Refer USD 316K-506KContinuous Delivery | Data Architecture | Data Engineering | Data Governance | Data LakeLearning and development programs | Remote workSenior-level Full TimeChicago, Illinois, USA R23h ago
-
Lead Data Engineer USD 316K-506KContinuous Delivery | Data Architecture | Data Engineering | Data Governance | Data LakesLearning and development programs | Mentoring and coaching | Remote workSenior-level Full TimeChicago, Illinois, USA R23h ago
-
Cloud Data Engineer USD 119K-208KAzure | Azure Data | Azure Data Factory | Azure SQL | Azure SynapseMid-level Full TimeIndianapolis, IN, US23h ago
-
Staff, Analytics Engineer, GTM Data Science & Analytics USD 155K-228KAutomated reconciliation | CI/CD | DBT | Data Modeling | Data ObservabilitySenior-level Full TimeRemote - US R23h ago
-
Data Architect (P4642) USD 162K-262KAI system architecture | Access Control | Audit Logging | Azure | Azure Data5 day in office scheduleSenior-level Full TimeCincinnati, OH; Chicago, IL1d ago
-
Senior Data Architect (AWS, Azure, GCP) USD 120K-200KAWS | Amazon DynamoDB | Amazon Kinesis | Apache Airflow | Apache Kafka1 to 1 care | 401k matching | Certification support | Disability insurance | Employee resource groupsSenior-level Full TimeReston, VA, United States R1d ago
-
Senior Data Architect (AWS, Azure, GCP) USD 120K-200KAWS | Amazon DynamoDB | Amazon Kinesis | Apache Airflow | Azure401k matching | Disability insurance | Employee resource groups | Fertility benefits | Health coverageSenior-level Full TimeRichmond, VA, United States R1d ago
-
Data Architect (AWS, Azure, GCP) USD 120K-200KAWS | Airflow | Azure | Cloud platform | Cosmos DB401k matching | Disability insurance | Employee wellness stipends | Fertility coverage | Flexible work environmentSenior-level Full TimeReston, VA, United States R1d ago
-
Data Architect (AWS, Azure, GCP) USD 120K-200KAWS | Airflow | Azure | CosmosDB | Data Governance401k matching | Disability insurance | Employee resource groups | Employer stipend benefits | Fertility coverageSenior-level Full TimeRichmond, VA, United States R1d ago