Lead Architect - Azure DBX
Tasks
- Architect Lakehouse solutions with Delta Lake and medallion architecture
- Build batch and real-time data processing solutions
- Design and implement scalable data platforms using Azure Databricks
- Develop ETL ELT pipelines with PySpark and Spark SQL
- Establish data quality monitoring governance and security frameworks
- Implement CI CD pipelines with Azure DevOps and Git source control
- Integrate data pipelines with Azure Data Factory Azure Data Lake Storage and Azure Synapse
- Lead data migration and modernization to Azure
- Mentor junior engineers and provide technical leadership
- Optimize Spark jobs cluster utilization partitioning and query performance
Perks/Benefits
- 401k plan
- Dental insurance
- Disability insurance
- Health insurance
- Life insurance
- PTO
- Paid Holidays
- Parental leave
- Vision insurance
Skills/Tech-stack
ARM | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2 | Azure Databricks | Azure DevOps | Azure Event | Azure Event Hubs | Azure SQL | Azure SQL Database | Azure Synapse | Azure Synapse Analytics | Bicep | CI/CD | Data Factory | Data Governance | Data Lake Storage Gen2 | Data Warehousing | Delta Lake | Delta Live Tables | Delta Live) | Dimensional Modeling | Event Hubs | Git | Infrastructure as Code | Kafka | Lakehouse | Metadata Management | PySpark | Python | RBAC | SQL Database | Spark SQL | Structured Streaming | Synapse Analytics | Terraform | Unity Catalog | “as-code”
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R11d ago
-
Software Engineer, Systems ML Engineering USD 170K-251KAlerting | Benchmarking | C++ | CUDA | DashboardingSenior-level Full TimeSunnyvale, CA | Bellevue, WA | …1h ago
-
ML Engineer, GenAI Ads, Search Personalization USD 207K-301KC++ | Clustering Algorithms | Data Processing | Data Structures | Data Structures and AlgorithmsSenior-level Full TimeMountain View, CA, USA1h ago
-
Customer Engineer, Data/Analytics, Google Public Sector USD 152K-222KApache Spark | Batch Processing | Big Data | Cloud Architecture | Data GovernanceBonus target | Equity | Health benefits | Paid time off | Retirement benefitsSenior-level Full TimeReston, VA, USA; Atlanta, GA, USA1h ago
-
Software Engineer, Product USD 208K-269KAgentic Frameworks | Anthropic | Claude Code | Cursor | DocumentationDental insurance | Health insurance | Paid time off | Retirement plan | Vision insuranceMid-level Full TimeSan Francisco, CA6h ago
-
Data Analysis | Data Visualization | GenAI | Java | Jupyter NotebookSenior-level Full TimeCupertino11h ago
-
Forward Deployed AI Engineer USD 180K-230KAPI | AWS | Azure | CI/CD | Cloud platform5-day workweek | Collaborative work culture | Flexible working hours | Supportive work environmentSenior-level Full TimeNew York, New York, United States12h ago
-
Principal AI Engineer USD 200K-245KAgent-based | Agent-based architecture | Amazon Web Services | Azure | CI/CD5 days per week | Collaborative culture | Flexible working hours | Supportive work environmentSenior-level Full TimeNew York, New York, United States12h ago
-
Machine Learning Engineer, Sensor Pipelines USD 175K-215KC++ | Computer Vision | JAX | Machine Learning | PyTorchHybrid work scheduleMid-level Full TimeMountain View, CA, USA12h ago
-
Sr. Embedded Software/Firmware Engineer USD 180K-240KARM | ASPICE | AUTOSAR | C# | C++Competitive compensation | Dental insurance | Flexible hours | Medical insurance | Pre-IPO stock optionsSenior-level Full TimeUnited States - Remote R12h ago
-
AI Engineer USD 105K-115KAPI Cost Modeling | API Integration | API Wrappers | Anthropic API | Conversation DesignCalm health employee membership | Company social events | Comprehensive benefits package | Employee referral incentive program | Employee resource groupsEntry-level Full TimeCincinnati, OH, US12h ago
-
Data Engineer USD 112K-160KAWS | AWS S3 | Airflow | Azure | CI/CD401k match | Family leave | Medical, dental, and vision insurance | PTO | Paid HolidaysMid-level Full TimeSanta Monica, CA, United States13h ago
-
Gen AI - CVS - Dallas USD 36K-126KAPI Gateway | Aiohttp | Amazon Web Services | Asyncio | CI/CD401k retirement plan | Medical, dental, vision benefits | Paid Holidays | Paid time off | Variable pay/incentivesMid-level Full TimeUnited States13h ago
-
Senior Numerical Optimization Engineer USD 191K-253KAPIs | C++ | CI/CD | CPLEX | Cloud ComputingComprehensive benefits | Health insurance | Paid on-call supportSenior-level Full TimeCosta Mesa, California, United States13h ago
-
Sr. Software Engineer, AI Infrastructure USD 144K-236KApache Beam | Apache Flink | Apache Spark | C# | C++Career development | Hybrid work | Stock grantsSenior-level Full TimeSunnyvale, CA, United States14h ago
-
Senior AI Engineer, Tools & Agents USD 200K-240KAPI Design | Agent Orchestration | Knowledge Base | LLM | Multi-tenantDental insurance | Flexible PTO | Health insurance | Parental leave | Visa supportSenior-level Full TimeSan Francisco R14h ago
-
Software Engineer, ML Platform (ML Training) USD 175K-265KAWS | Computer Vision | DeepSpeed | Distributed Training | JAXMid-level Full TimeFoster City, CA14h ago
-
Google Cloud Platform (GCP) Data Engineer USD 100K-203KApache Beam | BigQuery | Broadcast joins | CI/CD | CachingMid-level Full TimeArlington, VA14h ago
-
Google Cloud Platform (GCP) Data Engineer USD 100K-203KApache Beam | BigQuery | CI/CD | Cloud Storage | Cloud platformSecurity clearance providedMid-level Full TimeArlington, VA14h ago
-
Senior Data Engineer USD 180K-210KAirflow | Anomaly Detection | BigQuery | CI/CD | Cloud platform401k employer match | Annual Company Holidays | Catered lunches | Company retreat | Comprehensive health coverageSenior-level Full TimeNew York City14h ago
-
Senior Robotics Software Engineer USD 113K-151K3D simulation | AI | API Development | Agile | C++In-person office work | Lab workSenior-level Full TimeAustin, Texas, United States15h ago
-
Sr. Engineer, Firmware & Embedded Software USD 79K-158KAgile | Azure DevOps | BDM | Bare Metal | BootloaderOccasional travel for onboarding | Remote work within the United StatesSenior-level Full TimeFlorida, United States R15h ago
-
Senior Firmware Engineer, Space Embedded Systems USD 191K-253KARM | Analog Sensors | Bare Metal | Best practices | Board Bring-upSenior-level Full TimeCosta Mesa, California, United States15h ago
-
Firmware Engineer, Space Embedded Systems USD 166K-220KARM | Analog Sensors | Bare Metal | C# | C++Mid-level Full TimeCosta Mesa, California, United States15h ago
-
Senior Software Engineer; Analytics Infrastructure USD 167K-230KAPI Design | Backend Engineering | Caching | Columnar Databases | Data ModelingBack up child care support | Employee stock purchase program ESPP | Flexible time off | Home office & equipment stipend | Learning and development stipendSenior-level Full TimeSan Francisco, CA15h ago