Pyspark Data Engineer with Databricks
Tasks
- Build data ingestion and data modeling solutions
- Build orchestration workflows
- Design end-to-end ETL/ELT pipelines
- Develop and maintain Python PySpark data pipelines
- Develop and operationalize ML workflows with MLflow
- Implement CI CD for data and ML pipelines
- Implement data quality validation reconciliation anomaly detection
- Implement pipeline monitoring logging alerting observability
- Optimize Spark jobs for performance scalability cost
Perks/Benefits
- Employee assistance programs
- Life and disability insurance
- Medical, dental, and vision coverage
- Paid time off
- Retirement savings plan
Skills/Tech-stack
Airflow | Alerting | Anomaly Detection | Apache Spark | CI/CD | Cluster tuning | Data Modeling | Data Quality | Data Validation | Databricks | Databricks Workflows | Distributed Systems | ELT | ETL | Logging | MLflow | Monitoring | Orchestration | PySpark | Python | Snowflake | Spark optimization
Education
N/A
Roles
Regions
Countries
States
Cities
Related jobs
-
Data Engineer (remote) USD 85K-100KAgile | Apache Spark | Artificial Intelligence | Azure Data | Azure Data Factory401k match | Employee assistance program | Flexible schedule | Health insurance | Paid parental leaveMid-level Full TimeWork From Home, United States R4h ago
-
Software Engineer - Dragonfly Portfolio USD 160K-215KCryptography | Distributed Systems | Event Ingestion | Onchain Event Ingestion | Performance optimizationOnsite work locationMid-level Full TimeSan Francisco4h ago
-
Mid-level Full TimeSalt Lake City, Utah6h ago
-
Data Engineer ID50062 USD 148K-164KAWS | AWS SageMaker | AWS SageMaker Studio | Airflow | Apache SparkEducation budget | Fitness budget | Flexible schedule | Mentorship | Office optionsSenior-level Full TimeBlacksburg, United States6h ago
-
Full Stack Developer - Cloud Engineer USD 107K-160KAPI Design | Agile | Amazon SageMaker | Analytical Data | Analytical Data WarehouseMid-level Full TimeColumbus, Ohio, United States8h ago
-
Sr. Tech Lead, GTM Applied AI & Analytics USD 150K-243KAirflow | Data Warehousing | Databricks | Fine Tuning | LLM APIsSenior-level Full TimeSan Francisco, CA, United States8h ago
-
Data Engineer, Analytics USD 205K-235KData Governance | Data Modeling | Data Quality | Data Security | Data VisualizationEntry-level Full TimeSeattle, WA9h ago
-
Software Engineer, Machine Learning USD 185K-200KClassification | Computer Vision | Data Mining | Data Regression | Deep learningMid-level Full TimeMenlo Park, CA9h ago
-
Data Engineer (Analytics) USD 191K-235KBig Data | Data Modeling | Data Warehousing | Data integration | Dimensional ModelingDomestic and international travel | TelecommutingMid-level Full TimeMenlo Park, CA | Remote, US R9h ago
-
Robotics Manipulation Engineer USD 157K-240KAdaptive Control | C plus plus | Control Systems | Deep learning | GPUSenior-level Full TimeFremont, CA9h ago
-
Robotics Engineer - Logistics and Material Flow USD 170K-240KAGV | Automation | C++ | Cause analysis | Computer VisionTravel to data centers for engineering studiesSenior-level Full TimeFremont, CA9h ago
-
Ad Ranking | Algorithms | C++ | Data Processing | Data StructuresSenior-level Full TimeMountain View, CA, USA9h ago
-
Research Engineer, World Models, DeepMind USD 147K-211KAccelerator Training | C++ | Deep learning | Distributed Training | GPU ComputingMid-level Full TimeLondon, UK; New York, NY, USA9h ago
-
AI Engineer, Professional Services, Google Cloud USD 183K-265KApache Beam | Apache Spark | C++ | Data Validation | Data WarehousingTechnical workshops | Travel opportunitiesSenior-level Full TimeAustin, TX, USA; Atlanta, GA, USA9h ago
-
Senior Software Engineer, AI/ML GenAI, Core USD 174K-252KAlgorithms | C++ | Computer Vision | Data Processing | Data StructuresSenior-level Full TimeKirkland, WA, USA; Sunnyvale, CA, USA9h ago
-
Software Engineer III, AI/ML, Google Workspace USD 147K-211KC++ | Data Processing | Debugging | Language Processing | ML InfrastructureSenior-level Full TimeBoulder, CO, USA9h ago
-
Product Software Modernization Engineer, Quantum AI USD 147K-211KBazel | Cloud Spanner | Cloud Storage | Cloud platform | Distributed cloudMid-level Full TimeSeattle, WA, USA; Goleta, CA, USA9h ago
-
Software Engineer III, Infrastructure, GDC AI Storage USD 147K-211KCSI | Data Structures | Data Structures and Algorithms | Distributed Systems | GoSenior-level Full TimeKirkland, WA, USA9h ago
-
Automation | C++ | CSS | Database Design | HTMLMid-level Full TimeAnn Arbor, MI, USA9h ago
-
Software Engineering - DataStage Developer USD 112K-129KAxway | Axway Secure SFTP | Azure | Azure DevOps | CA7 SchedulerHybrid work schedule | Remote workMid-level Full TimeSyracuse, New York, United States12h ago
-
Mid-level Full TimeFlorida, United States13h ago
-
Senior Data Engineer - Knowledge Platform USD 160K-260KApache Airflow | Apache NiFi | Batch Processing | BigQuery | Cloud platformEquity compensation | Fully stocked kitchen | Open office space | Team building eventsSenior-level Full TimeUS - San Francisco17h ago
-
Robotics Platform Security Engineer USD 90K-300KAppArmor | Auditd | C# | C++ | CIS BenchmarksHybrid work option | On-site collaboration | Remote work optionSenior-level Full TimeIrvine, CA17h ago
-
Software Engineer II - Abnormal Data Platform USD 149K-214KAerospike | Amazon DynamoDB | Apache Spark | Data Storage | DatabricksDistributed team collaboration | Remote work | Technical mentorshipMid-level Full TimeRemote - USA R17h ago
-
Robotics Application & Product Security Engineer USD 90K-300KAPI Security | Adversarial analysis | Application Security | Artifact signing | AuthenticationHybrid or remote optionSenior-level Full TimeIrvine, CA17h ago