Lead Data Engineer (Databricks/Pyspark)
Tasks
- Apply unit tests CI CD pipelines
- Collaborate with business analysts
- Design scalable ETL pipelines
- Develop calculation engines
- Ensure data governance lineage audit compliance
- Ensure data quality and reconciliation
- Implement parameterized business rules
- Integrate with upstream/downstream systems
- Optimize Spark jobs
- Set up monitoring for production pipelines
Perks/Benefits
- Company-Paid Holidays
- Employee assistance programs
- Life and disability insurance
- Medical, dental, and vision coverage
- Paid time off
- Personal days
- Retirement savings plans
- Sick leave
Skills/Tech-stack
Airflow | Apache Spark | Bitbucket | CI/CD | Dagster | Data Modeling | Databricks | Databricks Workflows | Delta Lake | Delta Live Tables | Delta Live) | GitHub | Infrastructure as Code | Jenkins | Partitioning | Performance Tuning | PySpark | Query Performance Tuning | Query performance | SQL | Terraform | Unity Catalog | Workflow Orchestration | “as-code”
Education
N/A
Roles
Related jobs
-
Senior Data Engineer - Knowledge Platform USD 160K-260KApache Airflow | Apache NiFi | Batch Processing | BigQuery | Cloud platformEquity compensation | Fully stocked kitchen | Open office space | Team building eventsSenior-level Full TimeUS - San Francisco5h ago
-
Robotics Application & Product Security Engineer USD 90K-300KAPI Security | Adversarial analysis | Application Security | Artifact signing | AuthenticationHybrid or remote optionSenior-level Full TimeIrvine, CA5h ago
-
Machine Learning Engineer, Growth USD 130K-500KElasticsearch | Embeddings | Fine Tuning | Go | KafkaEquity grant | Free gym membership | Health insurance | Housing bonus | Meals stipendMid-level Full TimeSan Francisco8h ago
-
C++/CUDA Systems Engineer – Surgical Robotics Platform USD 140K-160KC++ | C++17 | C++20 | CPU GPU Scheduling | CUDAEquity | Health insurance | Paid time off | Performance bonusMid-level Full TimeLos Angeles, California10h ago
-
AWS Batch | AWS EC2 | AWS IAM | AWS Lambda | AWS S3Annual bonus | Company paid benefits | Equity | Paid time offSenior-level Full TimeLos Angeles, California11h ago
-
Principal Engineer, Data & ML Platform USD 119K-180KAPIs | Automated testing | Cloud Native | Cloud platform | Continuous DeploymentSenior-level Full TimeScottsdale, AZ11h ago
-
Sr Sales Engineer, West USD 160K-196KAnalytics | Apache Spark | Artificial Intelligence | Dataiku | Kubernetes401k match | Dental insurance | Employer paid disability coverage | Flexible spending accounts | Medical insuranceSenior-level Full TimeUnited States, Remote R11h ago
-
Data Engineer USD 129K-178KApache Kafka | Apache Spark | Cloud Platforms | Data Compliance | Data GovernanceMid-level Full TimeUS - Remote R12h ago
-
Sr. Data Engineer USD 145K-160KAPI Development | AWS | AWS Glue | AWS Lambda | Amazon AthenaComplimentary club membership | Personal training | Pilates | Shop discounts | SpaßSenior-level Full TimeNew York, NY, United States12h ago
-
Senior Machine Learning Engineer, Operations Research USD 173K-218KCombinatorial Optimization | Deep learning | Keras | Machine Learning | Mathematical ProgrammingAnnual refresh grants | Equity grant | Flexible work policy | Remote workSenior-level Full TimeUnited States - Remote R12h ago
-
Machine Learning Engineer USD 217K-260KAWS | Cloud platform | Dash | Database Management | Docker401k employer match | Comprehensive healthcare benefits | Family planning support | Flexible vacation | Gender-affirming careSenior-level Full TimeSan Francisco, CA14h ago
-
Machine Learning Engineer USD 260K-303KAmazon Web Services | Apache Beam | Apache Spark | Cassandra | Cloud platform401k employer match | Comprehensive healthcare benefits | Family planning support | Flexible vacation | Full-time Telecommuting OptionMid-level Full TimeNew York City, NY14h ago
-
Data/Analytics Engineering Co-op USD 67K-67KApache Airflow | Cloud infrastructure | Dagster | Looker | PowerBIHybrid workEntry-level Full TimeBoston or NYC14h ago
-
Data & Analytics Engineer USD 115K-150KCI/CD | CTE | Clustering | DBT | Data Lineage401k matching | Birthday time off | Cell phone reimbursement | Childcare expense reimbursement | Company-Paid HolidaysMid-level Full TimeUnited States14h ago
-
Storage Engineer USD 100K-120KAI code assistants | Ansible | Azure Blob | Azure Blob Storage | Azure Files401k matching | Bereavement | Employee assistance program | Employee discount program | Health and Dental/Vision CareMid-level Full TimeSt. Louis, MO, United States15h ago
-
Staff Analytics Engineer USD 138K-221KBigQuery | CI/CD | DBT | Data Governance | Data ModelingHybrid work modelSenior-level Full TimeBoston, MA15h ago
-
Staff Data Engineer USD 138K-221KAmazon Kinesis | Amazon Redshift | Amazon Web Services | Apache Airflow | BigQueryHybrid work modelSenior-level Full TimeBoston, MA16h ago
-
Principal Machine Learning & Data Engineer USD 184K-271KA/B | A/B Testing | AWS | Autoscaling | B testing401k | Healthcare | Paid parental leave | Paid sick time | Paid time offSenior-level Full TimeRemote - US R16h ago
-
Staff Site Reliability Engineer, Data Platform USD 131K-210KAccess Management | Automation | Azure | Azure Kubernetes | Azure Kubernetes ServiceSenior-level Full TimeAustin, TX, United States16h ago
-
Data Engineer (remote) USD 85K-100KAgile | Apache Hive | Apache Impala | Apache Spark | Artificial Intelligence401k match | Employee assistance program | Flexible schedule | Health insurance | Life insuranceMid-level Full TimeWork From Home, United States R16h ago
-
Senior Forward Deployed AI Engineer (Remote Eligible) USD 227K-245KAWS Bedrock | Agent Framework | Agent Orchestration | Cloud infrastructure | Databricks401k match | Dental insurance | Flexible time off | Health insurance | Life insuranceSenior-level Full Time-REMOTE, USA- R17h ago
-
Senior-level Full TimeHill AFB, UT17h ago
-
Data Engineer USD 125K-163KAWS | Apache Airflow | Azure | CI/CD | DBT401k match | Commuter benefits | Dental insurance | Disability insurance | Family planning benefitsSenior-level Full TimeNew York City17h ago
-
Senior Analytics Engineer USD 140K-170KAWS | Airflow | BigQuery | DBT | Data Validation401k match | Disability Leave | Family building benefits | Health benefits | Learning stipendSenior-level Full TimeGameChanger Remote - US R17h ago
-
Senior Data Engineer USD 138K-181KAWS | Apache Airflow | Apache Kafka | Apache Spark | Azure401k match | Commuter benefits | Dental insurance | Disability insurance | Fitness benefitsSenior-level Full TimeNew York City17h ago