Lead Data Engineer (Databricks/Pyspark)
Tasks
- Apply unit tests CI CD pipelines
- Collaborate with business analysts
- Design scalable ETL pipelines
- Develop calculation engines
- Ensure data governance lineage audit compliance
- Ensure data quality and reconciliation
- Implement parameterized business rules
- Integrate with upstream/downstream systems
- Optimize Spark jobs
- Set up monitoring for production pipelines
Perks/Benefits
- Company-Paid Holidays
- Employee assistance programs
- Life and disability insurance
- Medical, dental, and vision coverage
- Paid time off
- Personal days
- Retirement savings plans
- Sick leave
Skills/Tech-stack
Airflow | Apache Spark | Bitbucket | CI/CD | Dagster | Data Modeling | Databricks | Databricks Workflows | Delta Lake | Delta Live Tables | Delta Live) | GitHub | Infrastructure as Code | Jenkins | Partitioning | Performance Tuning | PySpark | Query Performance Tuning | Query performance | SQL | Terraform | Unity Catalog | Workflow Orchestration | “as-code”
Education
N/A
Roles
Related jobs
-
Senior Data Engineer - Knowledge Platform USD 160K-260KApache Airflow | Apache NiFi | Batch Processing | BigQuery | Cloud platformEquity compensation | Fully stocked kitchen | Open office space | Team building eventsSenior-level Full TimeUS - San Francisco6h ago
-
Software Engineer II - Abnormal Data Platform USD 149K-214KAerospike | Amazon DynamoDB | Apache Spark | Data Storage | DatabricksDistributed team collaboration | Remote work | Technical mentorshipMid-level Full TimeRemote - USA R6h ago
-
Robotics Application & Product Security Engineer USD 90K-300KAPI Security | Adversarial analysis | Application Security | Artifact signing | AuthenticationHybrid or remote optionSenior-level Full TimeIrvine, CA7h ago
-
Machine Learning Engineer, Growth USD 130K-500KElasticsearch | Embeddings | Fine Tuning | Go | KafkaEquity grant | Free gym membership | Health insurance | Housing bonus | Meals stipendMid-level Full TimeSan Francisco9h ago
-
Senior Analytics Engineer USD 87K-161KData Lakehouse | Data mesh | Databricks | Delta Lake | ETL401k | Health insurance | Hybrid work | Paid time off | Remote workSenior-level Full TimeRemote-MO, United States R10h ago
-
Data Engineer II USD 63K-114KData Processing | ETL | Real Time | Real-time Data | Real-time Data Processing401k | Flexible work schedule | Health insurance | Paid time off | Tuition reimbursementMid-level Full TimeRemote-MN, United States R10h ago
-
Lead Machine Learning Engineer USD 157K-237KA/B | A/B Testing | Airflow | B testing | Data PipelinesSenior-level Full TimeUS TX Austin12h ago
-
C++/CUDA Systems Engineer – Surgical Robotics Platform USD 140K-160KC++ | C++17 | C++20 | CPU GPU Scheduling | CUDAEquity | Health insurance | Paid time off | Performance bonusMid-level Full TimeLos Angeles, California12h ago
-
AWS Batch | AWS EC2 | AWS IAM | AWS Lambda | AWS S3Annual bonus | Company paid benefits | Equity | Paid time offSenior-level Full TimeLos Angeles, California12h ago
-
Principal Engineer, Data & ML Platform USD 119K-180KAPIs | Automated testing | Cloud Native | Cloud platform | Continuous DeploymentSenior-level Full TimeScottsdale, AZ12h ago
-
Sr Sales Engineer, West USD 160K-196KAnalytics | Apache Spark | Artificial Intelligence | Dataiku | Kubernetes401k match | Dental insurance | Employer paid disability coverage | Flexible spending accounts | Medical insuranceSenior-level Full TimeUnited States, Remote R13h ago
-
Data Engineer USD 129K-178KApache Kafka | Apache Spark | Cloud Platforms | Data Compliance | Data GovernanceMid-level Full TimeUS - Remote R13h ago
-
Sr. Data Engineer USD 145K-160KAPI Development | AWS | AWS Glue | AWS Lambda | Amazon AthenaComplimentary club membership | Personal training | Pilates | Shop discounts | SpaßSenior-level Full TimeNew York, NY, United States13h ago
-
Senior Machine Learning Engineer, Operations Research USD 173K-218KCombinatorial Optimization | Deep learning | Keras | Machine Learning | Mathematical ProgrammingAnnual refresh grants | Equity grant | Flexible work policy | Remote workSenior-level Full TimeUnited States - Remote R14h ago
-
Machine Learning Engineer USD 217K-260KAWS | Cloud platform | Dash | Database Management | Docker401k employer match | Comprehensive healthcare benefits | Family planning support | Flexible vacation | Gender-affirming careSenior-level Full TimeSan Francisco, CA15h ago
-
Machine Learning Engineer USD 260K-303KAmazon Web Services | Apache Beam | Apache Spark | Cassandra | Cloud platform401k employer match | Comprehensive healthcare benefits | Family planning support | Flexible vacation | Full-time Telecommuting OptionMid-level Full TimeNew York City, NY15h ago
-
Senior Data Engineer - Snowflake USD 165K-216KAWS | Azure | Cloud platform | Data Governance | Data IntegrityFlexible work arrangements | Onsite collaboration in Manhattan New York | Professional development opportunitiesSenior-level Full TimeNew York, New York, United States15h ago
-
Data/Analytics Engineering Co-op USD 67K-67KApache Airflow | Cloud infrastructure | Dagster | Looker | PowerBIHybrid workEntry-level Full TimeBoston or NYC15h ago
-
Data & Analytics Engineer USD 115K-150KCI/CD | CTE | Clustering | DBT | Data Lineage401k matching | Birthday time off | Cell phone reimbursement | Childcare expense reimbursement | Company-Paid HolidaysMid-level Full TimeUnited States15h ago
-
Storage Engineer USD 100K-120KAI code assistants | Ansible | Azure Blob | Azure Blob Storage | Azure Files401k matching | Bereavement | Employee assistance program | Employee discount program | Health and Dental/Vision CareMid-level Full TimeSt. Louis, MO, United States16h ago
-
Staff Analytics Engineer USD 138K-221KBigQuery | CI/CD | DBT | Data Governance | Data ModelingHybrid work modelSenior-level Full TimeBoston, MA17h ago
-
Staff Data Engineer USD 138K-221KAmazon Kinesis | Amazon Redshift | Amazon Web Services | Apache Airflow | BigQueryHybrid work modelSenior-level Full TimeBoston, MA17h ago
-
Principal Machine Learning & Data Engineer USD 184K-271KA/B | A/B Testing | AWS | Autoscaling | B testing401k | Healthcare | Paid parental leave | Paid sick time | Paid time offSenior-level Full TimeRemote - US R17h ago
-
Staff Site Reliability Engineer, Data Platform USD 131K-210KAccess Management | Automation | Azure | Azure Kubernetes | Azure Kubernetes ServiceSenior-level Full TimeAustin, TX, United States17h ago
-
Data Engineer (remote) USD 85K-100KAgile | Apache Hive | Apache Impala | Apache Spark | Artificial Intelligence401k match | Employee assistance program | Flexible schedule | Health insurance | Life insuranceMid-level Full TimeWork From Home, United States R17h ago