Pyspark Data Engineer with Databricks
Tasks
- Build data ingestion and data modeling solutions
- Build orchestration workflows
- Design end-to-end ETL/ELT pipelines
- Develop and maintain Python PySpark data pipelines
- Develop and operationalize ML workflows with MLflow
- Implement CI CD for data and ML pipelines
- Implement data quality validation reconciliation anomaly detection
- Implement pipeline monitoring logging alerting observability
- Optimize Spark jobs for performance scalability cost
Perks/Benefits
- Employee assistance programs
- Life and disability insurance
- Medical, dental, and vision coverage
- Paid time off
- Retirement savings plan
Skills/Tech-stack
Airflow | Alerting | Anomaly Detection | Apache Spark | CI/CD | Cluster tuning | Data Modeling | Data Quality | Data Validation | Databricks | Databricks Workflows | Distributed Systems | ELT | ETL | Logging | MLflow | Monitoring | Orchestration | PySpark | Python | Snowflake | Spark optimization
Education
N/A
Roles
Regions
Countries
States
Cities
Related jobs
-
Associate Data Engineer USD 46K-111KAutomated testing | Azure DevOps | BigQuery | CI/CD | Cloud StorageCompany-Paid Holidays | Employee assistance program | Life and disability insurance | Medical, dental, and vision coverage | Paid time offMid-level Full TimeNashville, TN, US6h ago
-
Analytics, Finance & Strategy USD 270K-320KAWS | Apache Airflow | Cloud platform | DBT | DashboardsFlexible working hours | Generous vacation | Optional equity donation matching | Parental leaveMid-level Full TimeSan Francisco, CA | New York …9h ago
-
Senior-level Full TimeDenver, Colorado, United States R9h ago
-
Robotics Infrastructure Engineer USD 94K-220KAlerting | Build Automation | CI runners | CI/CD | Camera ModelsHigh autonomy | High output environmentMid-level Full TimeWatertown, MA9h ago
-
Staff Data Analyst USD 154K-239KData Modeling | Data Pipelines | Data Quality | Data Transformation | Data VisualizationDiversity, equity, inclusion and belonging | Social impact | Well-being programsSenior-level Full TimeSan Francisco, California9h ago
-
Senior Software Engineer, AI/ML USD 142K-203KAgent systems | Algorithms | Anthropic API | Backend APIs | CI/CDSenior-level Full TimeAustin, Texas, United States9h ago
-
Agentic Analytics Engineer USD 186K-256KAI Agents | Airflow | Automation | BigQuery | DBT401k matching | Life insurance | Medical/Dental/Vision insurance | Unlimited PTOMid-level Full TimeSeattle, Washington, United States10h ago
-
Staff Machine Learning Engineer- AI Governance USD 169K-270KAI Governance | Agentic Frameworks | Bias | Data Drift | DockerSenior-level Full TimeFoster City, CA, United States10h ago
-
Senior-level Full TimeNew York, New York, United States11h ago
-
Senior-level Full TimeSan Francisco, California, United States11h ago
-
Senior Data Engineer Contractor- 8 months USD 120K-160KData Modeling | Data Partitioning | Data Quality | Data Validation | DatabricksSenior-level ContractWA - Bellevue11h ago
-
Senior Customer Success Engineer, North USD 144K-152KData Science | Databricks | Dataiku | Enterprise analytics | Generative AI401k company match | Commuter benefits | Dental insurance | Employer paid disability coverage | Flexible spending accountsSenior-level Full TimeUnited States, Remote R11h ago
-
Generative AI Consultant USD 94K-114KAWS | Anthropic | Azure | CI/CD | Chroma401k plan | Dental insurance | Flexible spending account | Flexible work environment | Gym reimbursementMid-level Full TimeNew York, NY, United States11h ago
-
Principal AI Engineer USD 115K-160KAPI Design | Agentic Systems | Artificial Intelligence | Backend Development | Data PipelinesBusiness travel insurance | Dental insurance | Disability insurance | Employee assistance program | Employee stock purchase planSenior-level Full TimeDallas, TX, United States12h ago
-
Senior Machine Learning Engineer, Personalization USD 184K-262KAWS | Apache Beam | Apache Spark | Cloud platform | Data Processing401k | Health insurance | Meal allowance | Paid flexible holidays | Paid parental leaveSenior-level Full TimeNew York, NY12h ago
-
Data Engineer USD 135K-220KAWS | Bash | Docker | Hadoop | JavaInsurance | Paid leave | TelecommutingEntry-level Full TimeChicago, United States R13h ago
-
Data Engineer (remote) USD 85K-100KAgile | Apache Spark | Artificial Intelligence | Azure Data | Azure Data Factory401k match | Employee assistance program | Flexible schedule | Health insurance | Paid parental leaveMid-level Full TimeWork From Home, United States R16h ago
-
Senior DataOps Engineer USD 141K-182KAWS CloudFormation | Amazon EMR | Amazon Web Services | Ansible | Apache AirflowEmployee assistance program | Flexible Time Away From Work | Health & medical benefits | Learning budget | Life and disabilitySenior-level Full TimeRemote US R16h ago
-
Software Engineer - Dragonfly Portfolio USD 160K-215KCryptography | Distributed Systems | Event Ingestion | Onchain Event Ingestion | Performance optimizationOnsite work locationMid-level Full TimeSan Francisco16h ago
-
Tech Lead, GTM Applied AI and Analytics USD 138K-225KAirflow | Amazon SageMaker | DBT | Databricks | Deep learningSenior-level Full TimeSan Francisco, CA, United States17h ago
-
Senior Software Engineer - Data Platform USD 186K-218KAirflow | Apache Kafka | Apache Spark | Caching | Cloud DataSenior-level Full TimeRemote - USA R17h ago
-
Forward Deployed Data Engineer USD 171K-269KAPI Integration | AWS | Authentication | Azure | Cloud PlatformsRemote work | Travel to customer sitesExecutive-level Full TimeWaltham, Massachusetts, United States17h ago
-
AI/ML Engineer USD 110K-165KAgile | Apache Spark | DevOps | Git | Machine Learning401k match | Dental insurance | Education & training benefits | Health insurance | Paid HolidaysMid-level Full TimeTampa, Hanscom, Colorado Springs, San Diego, … R18h ago
-
Mid-level Full TimeSalt Lake City, Utah18h ago
-
Data Engineer ID50062 USD 148K-164KAWS | AWS SageMaker | AWS SageMaker Studio | Airflow | Apache SparkEducation budget | Fitness budget | Flexible schedule | Mentorship | Office optionsSenior-level Full TimeBlacksburg, United States18h ago