Pyspark Data Engineer with Databricks
Tasks
- Build data ingestion and data modeling solutions
- Build orchestration workflows
- Design end-to-end ETL/ELT pipelines
- Develop and maintain Python PySpark data pipelines
- Develop and operationalize ML workflows with MLflow
- Implement CI CD for data and ML pipelines
- Implement data quality validation reconciliation anomaly detection
- Implement pipeline monitoring logging alerting observability
- Optimize Spark jobs for performance scalability cost
Perks/Benefits
- Employee assistance programs
- Life and disability insurance
- Medical, dental, and vision coverage
- Paid time off
- Retirement savings plan
Skills/Tech-stack
Airflow | Alerting | Anomaly Detection | Apache Spark | CI/CD | Cluster tuning | Data Modeling | Data Quality | Data Validation | Databricks | Databricks Workflows | Distributed Systems | ELT | ETL | Logging | MLflow | Monitoring | Orchestration | PySpark | Python | Snowflake | Spark optimization
Education
N/A
Roles
Regions
Countries
States
Cities
Related jobs
-
Automation Testing | CI/CD | CSS | Cypress | Feature DevelopmentMedical, dental & vision coverage | Paid time off | Parental leave | Reimbursement programs | Retirement planMid-levelRaleigh, United States R11d ago
-
Robotics Test & Data Engineer, Mapping USD 70K-300K3D Point Cloud Processing | 3D point cloud | Bash | Cloud processing | Cloud validationMid-level Full TimeIrvine, CA6h ago
-
Deployed Engineer (Seattle) USD 150K-250KAWS | Agent Orchestration | Containers | Demo Delivery | Failure handling401k plan | Dental insurance | Flexible vacation | Health insurance | Life insuranceMid-level Full TimeSeattle, WA11h ago
-
Systems Software Engineer - Robotic Platform USD 150K-200K*nix | C++ | Deterministic builds | Docker | LinuxSenior-level Full TimeAustin, Texas, United States12h ago
-
Software Engineer - Robotic Applications USD 160K-245KAutonomy Models | Behavior systems | C++ | Controls | LocalizationSenior-level Full TimeAustin, Texas, United States12h ago
-
Senior Developer, Data & IT - AI Solutions USD 120K-142KAI Agents | API Integration | AWS | AWS Bedrock | AWS SageMakerDental insurance | Dependent Care Account | Health insurance | Health savings account | Mental health counseling supportSenior-level Full TimeNew York, NY, United States12h ago
-
Senior Data Engineer USD 120K-142KApache Airflow | C# | Continuous integration | DBT | Data Modeling401b retirement savings program | Dental insurance | Dependent Care Account | Employer match | Flexible spending accountsSenior-level Full TimeNew York, NY, United States13h ago
-
Technical Support Engineer USD 90K-125KBusiness Intelligence | CTE | Data Modeling | Data Visualization | Database401k | Commuter benefits | Dog-friendly office | Equity | FSA benefitsSenior-level Full TimeSan Francisco, CA13h ago
-
Technical Support Engineer USD 90K-125KAWS | Amazon Redshift | Business Intelligence | CTE | Cloud Computing401k | Annual bonus | Commuter benefits | Dog-friendly office | EquitySenior-level Full TimeNew York City, NY13h ago
-
AI Research Engineer USD 190K-280KAgentic AI | Clinical data | Data Pipelines | Data integration | Deep learningDiversity and inclusion initiatives | Flexible work environment | Friendly work environment | Professional developmentMid-level Full TimeSeattle, Washington, United States; South San …14h ago
-
Principal AI Engineer - Nexus Black USD 135K-160KCI/CD | Cloud Native | Cloud Native Architecture | Distributed Systems | EvaluationHybrid workSenior-level Full TimeItasca, United States14h ago
-
Generative AI Inference Engineer USD 152K-287KAWS | CUDA | Cloud platform | Diffusion Models | DockerSenior-level Full TimeUnited States15h ago
-
Distinguished Software Engineer, Data Infrastructure USD 248K-406KAI Inference | AI Training | Batch Processing | Compliance | Data InfrastructureExecutive-level Full TimeMountain View, CA, United States15h ago
-
Senior Machine Learning Engineer USD 108K-160KA/B | A/B Testing | AWS Lambda | Algolia | Amazon OpenSearch401k | Dental insurance | Disability insurance | Life insurance | Medical insuranceSenior-level Full TimeRemote US R15h ago
-
Senior-level Full TimeMorristown, NJ, United States15h ago
-
Data Engineer I USD 85K-100KAWS Glue | AWS Lambda | Amazon Athena | Amazon Redshift | Amazon S3401k match | Direct student loan payment program | Employee recognition rewards | Flexible vacation | HRAMid-level Full TimeUnited States16h ago
-
Staff Data Engineer - Information Security USD 152K-248KAzure | Big Data | C# | C++ | Data PipelinesHealth and wellness programs | Time away from workSenior-level Full TimeSunnyvale, CA, United States16h ago
-
Manager, Data Engineering USD 130K-166KAWS | Access Controls | Apache Airflow | Audit Logging | AzureCollaborative team culture | Remote work | Work-life balanceSenior-level Full TimeRemote, United States R17h ago
-
Senior Software Engineer - Data Platform USD 150K-195KAPIs | BigQuery | C++ | CI/CD | Cloud Storage401k company contribution | Disability insurance | Fertility and infertility benefits | Industry competitive PTO | Learning and development opportunitiesSenior-level Full TimeUnited States R17h ago
-
AI Search | API Development | AWS | AWS Bedrock | Azure401k | Dental insurance | Medical insurance | Paid sick hours | Vision insuranceSenior-level Contract Full TimeRidgefield Park, NJ, United States17h ago
-
AWS S3 | Access Control | Active IQ | Ansible | Azure Blob Storage401k | Dental insurance | Medical insurance | Paid sick hours | Vision insuranceSenior-level Contract Full TimeRidgefield Park, NJ, United States17h ago
-
Senior Data Engineer (Remote, US) USD 140K-211KAirflow | Aiven | Apache Spark | BigQuery | CI/CD401k company contribution | Annual professional development fund | Be Well program | Paid volunteer service hours | Parental leaveSenior-level Full TimeRemote - United States R17h ago
-
Machine Learning Engineer USD 155K-228KAirflow | Amazon SageMaker | Apache Spark | Argo CD | Artificial IntelligenceHealthcare insurance | Paid parental leave | Paid personal time off | Paid sick time | Paid time offMid-level Full TimeRemote - US R19h ago
-
Prompt Engineer, Claude Code USD 300K-405KAI Safety | Behavioral evaluation | Prompt engineering | Python | System promptsFlexible working hours | Generous vacation | Health benefits | Optional equity donation matching | Parental leaveSenior-level Full TimeSan Francisco, CA | New York …19h ago
-
Staff Software Engineer, Data USD 270KApache Flink | Apache Kafka | Apache Spark | Cloud Data | Cloud data warehousing401k matching | ADND Insurance | Company holidays | Extended parental leave | Flexible spending accountSenior-level Full TimeUSA, Palo Alto19h ago