Lead Data Engineer (Databricks/Pyspark)
Tasks
- Apply unit tests CI CD pipelines
- Collaborate with business analysts
- Design scalable ETL pipelines
- Develop calculation engines
- Ensure data governance lineage audit compliance
- Ensure data quality and reconciliation
- Implement parameterized business rules
- Integrate with upstream/downstream systems
- Optimize Spark jobs
- Set up monitoring for production pipelines
Perks/Benefits
- Company-Paid Holidays
- Employee assistance programs
- Life and disability insurance
- Medical, dental, and vision coverage
- Paid time off
- Personal days
- Retirement savings plans
- Sick leave
Skills/Tech-stack
Airflow | Apache Spark | Bitbucket | CI/CD | Dagster | Data Modeling | Databricks | Databricks Workflows | Delta Lake | Delta Live Tables | Delta Live) | GitHub | Infrastructure as Code | Jenkins | Partitioning | Performance Tuning | PySpark | Query Performance Tuning | Query performance | SQL | Terraform | Unity Catalog | Workflow Orchestration | “as-code”
Education
N/A
Roles
Related jobs
-
Generative AI Consultant USD 105K-105KAWS | Azure | CI/CD | Deep learning | Docker401k matching | Health insurance | Paid time off | Parental leave | Professional developmentMid-level Full TimeSan Francisco, CA, United States4h ago
-
Senior Data Engineer USD 155K-180KAWS | Agile | Azure | Bash | Data Lakes401k savings plan | Commuter and parking tax savings benefit | Dental insurance | Employee assistance plan | Group medical insuranceSenior-level Full TimeBoston, United States12h ago
-
Data Architect USD 135K-165KAPI Integration | API Keys | API Management | Amazon Kinesis | Amazon Web ServicesHighly cleared team support | Security-focused environmentSenior-level Full TimeOther, VA13h ago
-
Mid-level Full TimeAuburn Hills, MI, United States14h ago
-
HR Analytics & Technology Engineer USD 110K-204KAlteryx | Artificial Intelligence | Data Governance | Data Modeling | Data Quality401k matching | Generous time off | Onsite fitness center | Tuition reimbursementSenior-level Full TimeUS-Rhode Island-Providence15h ago
-
Cause analysis | Cloud Computing | Data Enrichment | Data Monitoring | Data Processing401k match | Dental insurance | Health insurance | Life insurance | Paid HolidaysSenior-level Full TimeNew York15h ago
-
Deloitte Technology - US Associate GenAI Engineer USD 80K-100K.Net Core | AWS | Agile | Angular | AzureMid-level Full TimeHermitage, Tennessee, United States15h ago
-
Senior-level Full TimeDallas, Texas, United States15h ago
-
Kinetic Machine Learning Engineer USD 90K-123KClustering | Data Quality | Data quality profiling | Drift monitoring | Ensemble learning401k | Dental insurance | Educational assistance | Flexible savings account | Health savings accountMid-level Full TimeUnited States18h ago
-
AWS S3 | Almabase | Amazon Athena | Azure Data | Azure Data LakeMid-level Full TimeRemote within Texas, TX, US R19h ago
-
ALB | ALB/NLB | AWS | AWS CDK | AWS CloudSenior-level Full TimeDallas, Texas, United States1d ago
-
Statistics & Python Expert - Freelance AI Trainer USD 146K-146KCombinatorics | Graph theory | MATLAB | NumPy | Number theoryFlexible hours | Freelance opportunities | Project based workSenior-level FreelanceNew York, New York, United States … R1d ago
-
Statistics & Python Expert - Freelance AI Trainer USD 146K-146KC# | Combinatorics | Graph theory | MATLAB | NumPyFlexible hours | Part-time opportunities | Project based workSenior-level FreelanceTexas, United States - Remote R1d ago
-
Statistics & Python Expert - Freelance AI Trainer USD 146K-146KC# | MATLAB | NumPy | Pandas | PythonPart-time availability | Project based workSenior-level FreelanceMichigan, United States - Remote R1d ago
-
Technical Architect - Machine Learning USD 165K-200KAKS | AWS | Async Programming | Autogen | AzureSenior-level Full TimeUSA - Remote, United States R1d ago
-
Principal Big Data Software USD 178K-261K3GPP | 5G | Agile | Ansible | Apache NiFi401k plan | Adoption reimbursement | Employee assistance programs | Employee discounts | Employee wellness programsSenior-level Full TimeUSA:TX:Plano / W Plano Pkwy - …1d ago
-
Agile | CSS | Cloud Computing | DBT | Data LakesRelocation eligibleMid-level Full TimeUS-IN-INDIANAPOLIS-206 ~ 3939 Priority Way S …1d ago
-
Agile | CSS | DBT | Data Lakes | Data Modeling401k match | Dental insurance | Employee assistance program | Employee scholar program | Flexible spending accountsSenior-level Full TimeUS-IN-INDIANAPOLIS-206 ~ 3939 Priority Way S …1d ago
-
Staff Data Engineer USD 106K-284KAWS | Alerting | Amazon SNS | Apache Kafka | Argo CDMedical, dental, and vision coverage | Paid time off | Retirement savings options | Wellness programsSenior-level Full TimeWork At Home-Texas, United States1d ago
-
Mid-level Full TimeNew York, NY1d ago
-
Lead AI Software Engineer- AI Enablement USD 140K-231KAI Agents | AWS | AWS CDK | Agile Scrum | Amazon BedrockSenior-level Full TimeO'Fallon, Missouri (Main Campus), United States1d ago
-
Senior Machine Learning Engineer USD 150K-185KAWS | Artifact management | Auditability | Azure | Batch inferenceDental insurance | Disability insurance | EAP | Life insurance | Medical insuranceSenior-level Full TimeCHICAGO, IL, USA R1d ago
-
AWS | Agent Frameworks | Agent Orchestration | Amazon Kinesis | Amazon SageMaker401k plan | Dental insurance | Long-term disability | Medical insurance | Paid time offMid-level Full TimeCharlotte, 1120 S Tryon St., Corp, …1d ago
-
Agile | CI | Economic Dispatch | Extreme programming | GitAdoption Assistance | Employee assistance program | Health benefits | Paid parental leave | Relocation assistanceSenior-level Full TimeBellevue WA US, United States R1d ago
-
Quality Automation Engineer (SDET) - Career USD 95K-130KAirflow | Apache Beam | BigQuery | Bigtable | CI/CD401k matching | Career growth | Healthcare benefits | Online learning platform | Paid time offMid-level Full TimeUSA - Georgia - Alpharetta - …1d ago