Pyspark Data Engineer with Databricks
Tasks
- Build data ingestion and data modeling solutions
- Build orchestration workflows
- Design end-to-end ETL/ELT pipelines
- Develop and maintain Python PySpark data pipelines
- Develop and operationalize ML workflows with MLflow
- Implement CI CD for data and ML pipelines
- Implement data quality validation reconciliation anomaly detection
- Implement pipeline monitoring logging alerting observability
- Optimize Spark jobs for performance scalability cost
Perks/Benefits
- Employee assistance programs
- Life and disability insurance
- Medical, dental, and vision coverage
- Paid time off
- Retirement savings plan
Skills/Tech-stack
Airflow | Alerting | Anomaly Detection | Apache Spark | CI/CD | Cluster tuning | Data Modeling | Data Quality | Data Validation | Databricks | Databricks Workflows | Distributed Systems | ELT | ETL | Logging | MLflow | Monitoring | Orchestration | PySpark | Python | Snowflake | Spark optimization
Education
N/A
Roles
Regions
Countries
States
Cities
Related jobs
-
Automation Testing | CI/CD | CSS | Cypress | Feature DevelopmentMedical, dental & vision coverage | Paid time off | Parental leave | Reimbursement programs | Retirement planMid-levelRaleigh, United States R11d ago
-
Robotics Test & Data Engineer, Mapping USD 70K-300K3D Point Cloud Processing | 3D point cloud | Bash | Cloud processing | Cloud validationMid-level Full TimeIrvine, CA4h ago
-
Senior Developer, Data & IT - AI Solutions USD 120K-142KAI Agents | API Integration | AWS | AWS Bedrock | AWS SageMakerDental insurance | Dependent Care Account | Health insurance | Health savings account | Mental health counseling supportSenior-level Full TimeNew York, NY, United States11h ago
-
Senior Data Engineer USD 120K-142KApache Airflow | C# | Continuous integration | DBT | Data Modeling401b retirement savings program | Dental insurance | Dependent Care Account | Employer match | Flexible spending accountsSenior-level Full TimeNew York, NY, United States11h ago
-
Technical Support Engineer USD 90K-125KBusiness Intelligence | CTE | Data Modeling | Data Visualization | Database401k | Commuter benefits | Dog-friendly office | Equity | FSA benefitsSenior-level Full TimeSan Francisco, CA12h ago
-
Technical Support Engineer USD 90K-125KAWS | Amazon Redshift | Business Intelligence | CTE | Cloud Computing401k | Annual bonus | Commuter benefits | Dog-friendly office | EquitySenior-level Full TimeNew York City, NY12h ago
-
AI Research Engineer USD 190K-280KAgentic AI | Clinical data | Data Pipelines | Data integration | Deep learningDiversity and inclusion initiatives | Flexible work environment | Friendly work environment | Professional developmentMid-level Full TimeSeattle, Washington, United States; South San …12h ago
-
Principal AI Engineer - Nexus Black USD 135K-160KCI/CD | Cloud Native | Cloud Native Architecture | Distributed Systems | EvaluationHybrid workSenior-level Full TimeItasca, United States13h ago
-
Generative AI Inference Engineer USD 152K-287KAWS | CUDA | Cloud platform | Diffusion Models | DockerSenior-level Full TimeUnited States13h ago
-
Distinguished Software Engineer, Data Infrastructure USD 248K-406KAI Inference | AI Training | Batch Processing | Compliance | Data InfrastructureExecutive-level Full TimeMountain View, CA, United States13h ago
-
Senior Machine Learning Engineer USD 108K-160KA/B | A/B Testing | AWS Lambda | Algolia | Amazon OpenSearch401k | Dental insurance | Disability insurance | Life insurance | Medical insuranceSenior-level Full TimeRemote US R13h ago
-
Senior-level Full TimeMorristown, NJ, United States14h ago
-
Data Engineer I USD 85K-100KAWS Glue | AWS Lambda | Amazon Athena | Amazon Redshift | Amazon S3401k match | Direct student loan payment program | Employee recognition rewards | Flexible vacation | HRAMid-level Full TimeUnited States14h ago
-
Staff Data Engineer - Information Security USD 152K-248KAzure | Big Data | C# | C++ | Data PipelinesHealth and wellness programs | Time away from workSenior-level Full TimeSunnyvale, CA, United States15h ago
-
Manager, Data Engineering USD 130K-166KAWS | Access Controls | Apache Airflow | Audit Logging | AzureCollaborative team culture | Remote work | Work-life balanceSenior-level Full TimeRemote, United States R15h ago
-
Senior Software Engineer - Data Platform USD 150K-195KAPIs | BigQuery | C++ | CI/CD | Cloud Storage401k company contribution | Disability insurance | Fertility and infertility benefits | Industry competitive PTO | Learning and development opportunitiesSenior-level Full TimeUnited States R15h ago
-
AI Search | API Development | AWS | AWS Bedrock | Azure401k | Dental insurance | Medical insurance | Paid sick hours | Vision insuranceSenior-level Contract Full TimeRidgefield Park, NJ, United States16h ago
-
AWS S3 | Access Control | Active IQ | Ansible | Azure Blob Storage401k | Dental insurance | Medical insurance | Paid sick hours | Vision insuranceSenior-level Contract Full TimeRidgefield Park, NJ, United States16h ago
-
Senior Data Engineer (Remote, US) USD 140K-211KAirflow | Aiven | Apache Spark | BigQuery | CI/CD401k company contribution | Annual professional development fund | Be Well program | Paid volunteer service hours | Parental leaveSenior-level Full TimeRemote - United States R16h ago
-
Machine Learning Engineer USD 155K-228KAirflow | Amazon SageMaker | Apache Spark | Argo CD | Artificial IntelligenceHealthcare insurance | Paid parental leave | Paid personal time off | Paid sick time | Paid time offMid-level Full TimeRemote - US R17h ago
-
Prompt Engineer, Claude Code USD 300K-405KAI Safety | Behavioral evaluation | Prompt engineering | Python | System promptsFlexible working hours | Generous vacation | Health benefits | Optional equity donation matching | Parental leaveSenior-level Full TimeSan Francisco, CA | New York …17h ago
-
Staff Software Engineer, Data USD 270KApache Flink | Apache Kafka | Apache Spark | Cloud Data | Cloud data warehousing401k matching | ADND Insurance | Company holidays | Extended parental leave | Flexible spending accountSenior-level Full TimeUSA, Palo Alto17h ago
-
Sr Data Quality Engineer - SDET USD 140KAPI Testing | Apache Airflow | Cloud infrastructure | Data Lineage | Data Pipelines401k | Catered lunches | Commuter benefits | Continued education benefit | Equity compensationSenior-level Full TimeNew York, New York, United States18h ago
-
Sr Data Quality Engineer - SDET USD 140KAPI Testing | Airflow | Alerting | Cloud Databases | Data Lineage401k | Catered lunches | Commuter benefits | Continued education benefit | FSASenior-level Full TimeLos Angeles, California, United States18h ago
-
Sr Data Quality Engineer - SDET USD 140KAPI Testing | Airflow | Cloud Databases | Data Lineage | Data Quality401k | Catered lunches | Commuter benefits | Continued education benefit | FSASenior-level Full TimeSan Francisco, California, United States18h ago