Lead Data Engineer - Databricks Architect
Tasks
- Architect data storage solutions using ADLS Gen2 and data product containers
- Build and govern medallion data layers for analytics and AI use cases
- Collaborate with global teams and stakeholders to deliver data products
- Define and enforce data architecture standards reference architectures and design patterns
- Design and oversee data governance security and access control models using Unity Catalog
- Design data acquisition frameworks using CDC batch micro-batch and real-time streaming with Spark
- Integrate CI CD pipelines with GitHub workflows and automated deployment
- Lead data and AI platform architecture on Azure
- Mentor engineers and architects
- Provide hands-on leadership with Python PySpark and Databricks Notebooks
- Use AI code assistance tools to improve code quality
Perks/Benefits
- Company-Paid Holidays
- Employee assistance programs
- Life and disability insurance
- Medical, dental, and vision coverage
- Paid time off
- Retirement savings plan
Skills/Tech-stack
Access Control | Azure | Azure Data | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2 | Batch Processing | CDC | CI/CD | Data Governance | Data Lake Storage Gen2 | Data Security | Databricks | Databricks Notebooks | GitHub | GitHub Actions | Medallion Architecture | Micro-batch | Micro-batch processing | PySpark | Python | Real Time | Real-time Streaming | Spark | Structured data | Unity Catalog | Unstructured Data
Education
N/A
Related jobs
-
Senior Data Engineer - Knowledge Platform USD 160K-260KApache Airflow | Apache NiFi | Batch Processing | BigQuery | Cloud platformEquity compensation | Fully stocked kitchen | Open office space | Team building eventsSenior-level Full TimeUS - San Francisco4h ago
-
Robotics Platform Security Engineer USD 90K-300KAppArmor | Auditd | C# | C++ | CIS BenchmarksHybrid work option | On-site collaboration | Remote work optionSenior-level Full TimeIrvine, CA5h ago
-
Robotics Application & Product Security Engineer USD 90K-300KAPI Security | Adversarial analysis | Application Security | Artifact signing | AuthenticationHybrid or remote optionSenior-level Full TimeIrvine, CA5h ago
-
Machine Learning Engineer, Growth USD 130K-500KElasticsearch | Embeddings | Fine Tuning | Go | KafkaEquity grant | Free gym membership | Health insurance | Housing bonus | Meals stipendMid-level Full TimeSan Francisco7h ago
-
Abuse Test Engineer, Energy Storage USD 140K-224KActuation | Data Analysis | Data acquisition | Data logging | Electrochemical systemsSenior-level Full TimeMcCarran, NV9h ago
-
C++/CUDA Systems Engineer – Surgical Robotics Platform USD 140K-160KC++ | C++17 | C++20 | CPU GPU Scheduling | CUDAEquity | Health insurance | Paid time off | Performance bonusMid-level Full TimeLos Angeles, California10h ago
-
AWS Batch | AWS EC2 | AWS IAM | AWS Lambda | AWS S3Annual bonus | Company paid benefits | Equity | Paid time offSenior-level Full TimeLos Angeles, California10h ago
-
Staff Applied AI Engineer, Enterprise GenAI USD 216K-270KAWS | Cloud platform | Data Analysis | Generative AI | Google CloudCommuter stipend | Equity compensation | Health, dental, vision insurance | Learning and development stipend | Paid time offSenior-level Full TimeSan Francisco, CA; Seattle, WA; New …10h ago
-
AI/ML Engineer USD 130K-223KAgentic AI | Deep learning | Distributed Training | Docker | EmbeddingsMid-level Full TimeScottsdale, AZ11h ago
-
Principal Engineer, Data & ML Platform USD 119K-180KAPIs | Automated testing | Cloud Native | Cloud platform | Continuous DeploymentSenior-level Full TimeScottsdale, AZ11h ago
-
AI Ops Specialist USD 150K-210KAPI Integration | Automation | Evaluation | GitHub API | LLM Agents401k match | Dental insurance | Hardware setup | Health insurance | Unlimited PTOMid-level Full TimeNew York11h ago
-
Data Engineer USD 129K-178KApache Kafka | Apache Spark | Cloud Platforms | Data Compliance | Data GovernanceMid-level Full TimeUS - Remote R11h ago
-
Machine Learning Engineer, Foundation Model USD 129K-247KAuto-regressive models | C plus plus | Deep learning | Diffusion Models | Distributed TrainingSenior-level Full TimeSan Jose11h ago
-
Mid-level Full TimeLondon, New York City12h ago
-
AI Engineer USD 53K-119KAPI Design | Cost Optimization | Embeddings | Evaluation | JSONDental insurance | Gym stipend | Health insurance | Medical membership | Offsite retreatsSenior-level Full TimeRemote, US R12h ago
-
Sr. Data Engineer USD 145K-160KAPI Development | AWS | AWS Glue | AWS Lambda | Amazon AthenaComplimentary club membership | Personal training | Pilates | Shop discounts | SpaßSenior-level Full TimeNew York, NY, United States12h ago
-
Distinguished Software Engineer, Data Infrastructure USD 248K-406KAI | Batch Processing | Data Infrastructure | Data Privacy | Data ProcessingExecutive-level Full TimeMountain View, CA, United States12h ago
-
Senior Machine Learning Engineer, Operations Research USD 173K-218KCombinatorial Optimization | Deep learning | Keras | Machine Learning | Mathematical ProgrammingAnnual refresh grants | Equity grant | Flexible work policy | Remote workSenior-level Full TimeUnited States - Remote R12h ago
-
Machine Learning & Game Tech Architect USD 186K-257KAPI Design | C# | C++ | Game engines | Go401k matching | Dental insurance | Flexible schedule | Flexible working hours | Health insuranceSenior-level Full TimeBoston, MA, United States R13h ago
-
Machine Learning Engineer USD 217K-260KAWS | Cloud platform | Dash | Database Management | Docker401k employer match | Comprehensive healthcare benefits | Family planning support | Flexible vacation | Gender-affirming careSenior-level Full TimeSan Francisco, CA13h ago
-
Senior Software Engineer, Machine Learning USD 216K-303KDjango | GNN | Graph Neural Networks | JavaScript | Keras401k with employer match | Comprehensive healthcare benefits | Family planning support | Flexible vacation | Gender-affirming careSenior-level Full TimeSan Francisco, CA13h ago
-
Machine Learning Engineer USD 260K-303KAmazon Web Services | Apache Beam | Apache Spark | Cassandra | Cloud platform401k employer match | Comprehensive healthcare benefits | Family planning support | Flexible vacation | Full-time Telecommuting OptionMid-level Full TimeNew York City, NY13h ago
-
A/B | A/B Testing | AUC | B testing | CTR401k match | Commuter benefits | Dental insurance | FSA | Flexible work scheduleSenior-level Full TimeSan Francisco, CA13h ago
-
Data/Analytics Engineering Co-op USD 67K-67KApache Airflow | Cloud infrastructure | Dagster | Looker | PowerBIHybrid workEntry-level Full TimeBoston or NYC13h ago
-
Data & Analytics Engineer USD 115K-150KCI/CD | CTE | Clustering | DBT | Data Lineage401k matching | Birthday time off | Cell phone reimbursement | Childcare expense reimbursement | Company-Paid HolidaysMid-level Full TimeUnited States14h ago