Senior Data Engineer, Databricks
Tasks
- Build and optimize Databricks production pipelines
- Build semantic models with metadata and business definitions
- Design and implement Unity Catalog governance
- Design lakehouse medallion layers and Delta tables
- Drive data ingestion from ERP and operational systems
- Enable self service analytics with curated trusted data
- Ensure data reliability with quality and monitoring
- Implement CI CD across Databricks environments
- Integrate Power BI serving layer with gold layer
- Migrate legacy SQL Server and Synapse workloads
- Orchestrate ingestion and transformations in Databricks
- Own data platform architecture for ERDW.Next
- Review and verify AI generated coding outputs
- Tune performance and manage data platform costs
Perks/Benefits
- 401k company match
- Accrued vacation time off
- Dental insurance
- Long-term disability
- Medical insurance
- Paid Holidays
- Paid parental leave
- Paid sick leave
- Short-term disability
- Supplemental life insurance
- Term life insurance
- Vision insurance
Skills/Tech-stack
Apache Spark | Asset bundles | Auto Loader | Azure Data | Azure Data Factory | Azure DevOps | CDC | CI/CD | Conformed Dimensions | Data Factory | Data Governance | Data Modeling | Data Quality | Databricks | Databricks (asset) | Databricks Asset Bundles | Databricks Workflows | Delta Lake | Delta Live Tables | Delta Live) | Git | Lineage | Liquid clustering | Merge | Monitoring | Optimize | Photon | Power BI | PySpark | SSRS | Semantic Modeling | Slowly Changing Dimensions | Spark SQL | Star Schema | Synapse | T-SQL | Unity Catalog | ZORDER
Education
Roles
Related jobs
-
Lead Machine Learning Engineer USD 139K-304KAI Services | Agile | Azure AI | Azure AI Services | Azure Container401k matching | Adoption Assistance | Childcare tuition discounts | Company bonus | Fertility benefitsSenior-level Full TimeAnn Arbor, MI, United States14h ago
-
Senior Data Analytics Engineer USD 175K-214KBusiness Intelligence | Dashboard Development | Data Engineering | Data Pipelines | Data QualitySenior-level Full TimeSan Francisco, CA1d ago
-
Senior-level Full TimePalo Alto, CA1d ago
-
Principal Data Engineer (Python, AWS, Redshift, EMR, Airflow, Databricks, Big data) | Contract to Hire | Irvine, CA (Hybrid) | A USD 147K-170KAWS RDS | Airflow | Amazon EMR | Amazon Redshift | Amazon Web ServicesContract-to-hire | Hybrid work | Mentorship opportunitiesSenior-level Full TimeIrvine, CA R1d ago
-
Senior Data Engineer ID71671 USD 155K-170KAWS Kinesis | Apache Airflow | Apache Kafka | Automated testing | CI/CDAnnual learning budget | Flexible hours | Internal tech talks | Mentorship | People focused supportSenior-level Full TimeFort Lauderdale, United States1d ago
-
Senior Data Engineer ID71671 USD 155K-170KAWS Kinesis | Apache Airflow | Apache Kafka | Automated testing | CI/CDFlexible hours | Learning budget | Mentorship | People focused support | Remote workSenior-level Full TimeBaltimore, United States1d ago
-
Senior Data Engineer ID71671 USD 155K-170KAWS | AWS Kinesis | Apache Airflow | Apache Kafka | Automated testingFlexible hours | Internal TechTalks | Learning budget | Mentorship | Performance reviewsSenior-level Full TimePort Charlotte, United States1d ago
-
Senior Data Engineer ID71671 USD 155K-170KAmazon Web Services | Apache Airflow | Apache Kafka | Automated testing | Batch ProcessingFlexible hours | Internal TechTalks | Learning budget | Mentorship | No micromanagementSenior-level Full TimeAustin, United States1d ago
-
Databricks Engineer USD 175K-250KAmazon Web Services | Apache Spark | Azure | Azure Data | Azure Data FactorySenior-level Full TimeMilpitas, United States1d ago
-
Project Data Scientist and Analytics USD 125K-208KAPI Development | APIs | Cloud Computing | Dashboards | Data ArchitectureSenior-level Full TimeSan Diego, California, United States1d ago
-
Principal Machine Learning Engineer USD 172K-250KApache Arrow | Apache Spark | Bias Mitigation | DPO | Data-parallelSenior-level Full TimeUnited States R2d ago
-
Data Engineer/Sr Data Engineer, IT Analytics USD 38K-55KAgile | Apache Spark | Azure Data | Azure Data Factory | Azure Data Lake401 k program | Dental insurance | Discounts | Employee assistance program | Health benefitsSenior-level Full TimeFort Worth, TX, US2d ago
-
Senior Power BI Software Engineer - Clearance Required USD 166K-309KBI Desktop | Containerization | DAX | Database Design | Database optimization401k match | EAP | Education assistance | Flexible spending accounts | Flexible work arrangementsSenior-level Full TimeAnnapolis Junction, MD, US2d ago
-
Embedded Software Engineer Stf - E4 USD 120K-224KAgile | Atlassian | C# | C++ | CI/CD401k match | EAP | Education assistance | Flexible work schedule | HolidaysMid-level Full TimeGrand Prairie, TX, US2d ago
-
Staff Machine Learning Engineer USD 165K-236KClass imbalance | Docker | Error Analysis | Feature Engineering | Feature Labeling401k | Bonus | Equity | Generous vacation time | Medical/Dental/Vision insuranceSenior-level Full TimeRemote USA R2d ago
-
Staff Data Engineer USD 135K-180KAI coding | AI coding tools | AWS EKS | Alerting | Amazon EC2401k match | Career development | Parental leaveSenior-level Full TimeTempe, Arizona R2d ago
-
AWS | Azure | CI/CD | Container Networking | Container networking interface401k company match | ESPP | Family leave programs | Flexible spending accounts | Flexible time away planSenior-level Full TimeSan Diego, CALIFORNIA, United States R2d ago
-
AWS | Azure | CI/CD | Container Networking | Containers401k match | ESPP | Family leave programs | Flexible spending accounts | Flexible time awaySenior-level Full TimeSanta Clara, California, United States R2d ago
-
Principal Software Engineer- Embedded OS USD 125K-150KAgile | Artifactory | Atlassian Bamboo | Atlassian Confluence | BSPCareer growth opportunities | Comprehensive benefits | MentorshipSenior-level Full TimeDenver, Colorado, United States2d ago
-
Senior-level Full TimeDenver, CO - Hybrid; New York, … R2d ago
-
Staff+ Software Engineer, Safeguards Data USD 320K-485KAWS | Access Control | Cloud platform | Data Governance | Data LineageFlexible working hours | Generous vacation | Hybrid office policy | Optional equity donation matching | Parental leaveSenior-level Full TimeSan Francisco, CA | New York … R2d ago
-
Specialist Solutions Architect - AI/ML USD 180K-247KAI guardrails | AWS | Agent Orchestration | Artificial Intelligence | AzureSenior-level Full TimeUnited States2d ago
-
Senior Robotics Software Engineer, Generalist USD 220K-292KAgent systems | Autonomy | Behavior Trees | C++ | Computer VisionHealth benefits | Recovery Benefits | Travel for testing and demonstrationsSenior-level Full TimeBoston, Massachusetts, United States R2d ago
-
Senior Database Engineer USD 130K-150KAWS | AWS CDK | AWS CloudFormation | Access Control | Amazon AuroraSenior-level Full TimeLas Vegas, Nevada, United States; Seattle, … R2d ago
-
Embedded Software Engineer USD 100K-135KAgile | Bare Metal | Bash | C# | C++401k match | Employee support resources | Flexible Rewards Program | Flexible time off | Gym membershipsMid-level Full TimePittsburgh, PA2d ago