Data Engineer
Tasks
- Build and tune Apache Spark jobs for large scale processing
- Build entity resolution and record linkage logic
- Collaborate on semantic models and datasets for dashboards
- Design build and operate batch and real time data pipelines
- Establish and enforce data governance with Unity Catalog
- Implement CI CD for data and infrastructure
- Implement automated data quality validation profiling and anomaly detection
- Implement medallion architecture with Delta Lake and Delta Live Tables
- Ingest and process high volume IoT and historian data
- Model analytics ready datasets and serving layers
- Provide governed feature ready data for ML workflows
- Set up monitoring logging and observability for pipelines
- Support production incidents and create runbooks
Perks/Benefits
- 401k match
- Dental insurance
- Flexible paid time off
- Health insurance
- Life insurance
- Long-term disability
- Paid parental leave
- Short-term disability
- Vision insurance
Skills/Tech-stack
Airflow | Apache Spark | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DevOps | CI/CD | DBT | Data Factory | Data Governance | Data Lake Storage | Data Modeling | Data Quality | Databricks | Delta Lake | Delta Live Tables | Delta Live) | Docker | Entity Resolution | Event Hubs | GitHub Actions | Infrastructure as Code | Kafka | Logging | Monitoring | Observability | Power BI | PySpark | Python | Record Linkage | SQL | Snowflake | Spark SQL | Spotfire | Structured Streaming | Terraform | Unity Catalog | “as-code”
Roles
Related jobs
-
Associate Director, AI Enablement & Machine Learning USD 212K-318KCloud Computing | Data Engineering | Data Governance | Deep learning | Generative AIAnnual incentive program | Healthcare coverage | Retirement benefitsMid-level Full TimeCambridge, MA10h ago
-
Software Engineer, Data (L1) USD 75K-130KAI Platform | APIs | Apache Airflow | Apache Iceberg | Apache Spark401k match | Calm subscription | Career growth and learning opportunities | Commuter benefits options | Dental insuranceMid-level Full TimeAustin, TX12h ago
-
Solutions Engineer – Robotics & Autonomous Driving USD 152K-500K2D Imaging | 3D Imaging | Autonomous Driving | C++ | CameraCommuter stipend | Generous PTO | Global travel | Health, dental, vision coverage | Learning and development stipendSenior-level Full TimeSan Francisco, CA; New York, NY13h ago
-
Senior Software Engineer, Storage USD 217K-303KC++ | Caching | Cassandra | Data Storage | Distributed Systems401k employer match | Dental insurance | Equity compensation | Generous time off | Medical insuranceSenior-level Full TimeRemote - United States R15h ago
-
Machine Learning Research Engineer USD 161K-189KAWS | Azure | Bias Variance | Bias-Variance Tradeoff | C++Flexible hybrid work model | Mental health counseling | Mentorship programs | Paid parental leave | Paid volunteer time offMid-level Full TimeNew York, US, New York16h ago
-
Big Data | Cloud Cost Optimization | Cost Optimization | Data Governance | Data Security401k plan | Dental insurance | Health insurance | Life insurance | Paid time offMid-level Full TimeSpringdale, AR Terminal Office - Springdale, …16h ago
-
Senior Data Engineer - Logistics USD 38K-38KBig Data | Cloud Cost Optimization | Column-Level Security | Cost Optimization | Data Governance401k plan | Dental insurance | Health insurance | Life insurance | Paid time offSenior-level Full TimeSpringdale, AR Terminal Office - Springdale, …16h ago
-
Sr. Application Software Engineer, Data Analytics USD 160K-225KAngular | C# | CI/CD | Computer Vision | Continuous integrationExtended hours | Travel | Weekend workSenior-level Full TimeBastrop, TX17h ago
-
AI Engineer - Automation/Robotics USD 104K-110KAlgorithms | Azure Machine Learning | Computer Vision | Data Structures | DebuggingOnsite workEntry-level Full TimeRedmond, WA17h ago
-
Mid-level Full TimeEl Segundo18h ago
-
Senior-level Full TimeSan Francisco, California18h ago
-
Foundation Model Data, Software Engineer USD 213K-263KApache Beam | Apache Flume | C++ | CI/CD | Distributed Systems401k match | Baby bonding leave | Dental insurance | Disability insurance | Health insuranceSenior-level Full TimeMountain View, CA, USA; San Francisco, …18h ago
-
Senior-level Full TimeOnsite - Austin, TX18h ago
-
Machine Learning Engineer USD 196K-196KApache Airflow | CI/CD | Containerization | Docker | Experiment trackingFully remoteSenior-level Full TimeSan Francisco, CA R18h ago
-
Data parallelism | Diffusion Models | Efficient Attention | Expert parallelism | FlaxSenior-level Full TimeMountain View, California, United States, New …18h ago
-
Lead Cyber Risk & Analytics Engineer USD 130K-160KAgile | Data Visualization | Database querying | Econometrics | ExcelDependent care assistance | Flexible hours | Hybrid work | Learning and development | Mentorship programsSenior-level Full TimeNew York Office18h ago
-
Software Engineer - Embedded Firmware USD 125K-210KAvionics | C Programming | C++ | Communications Systems | DebuggingDental insurance | Equity compensation | Medical insurance | Paid time off | Performance bonusesSenior-level Full TimeSouth San Francisco, California, USA18h ago
-
AWS Bedrock | Agent systems | Anthropic API | Autogen | Azure401k matching program | Adoption Assistance | Development and career growth opportunities | Fertility treatments | Flexible work schedulesSenior-level Contract Full TimeRemote, OR, United States R19h ago
-
Senior Data Engineer USD 90K-110KAWS | Agile | Apache NiFi | Data Architecture | Data ModelingAutonomy | Flexible working hours | Global employee assistance programme | Online training videos | Teambuilding eventsSenior-level Full TimeNew York, United States19h ago
-
Data Platform Engineer USD 85K-120KAlembic | App Service | Azure | Azure App | Azure App Service401k match | Annual discretionary bonus program | Bereavement leave | Commuter spending accounts | Dental insuranceMid-level Full TimeChicago, IL (Downtown)19h ago
-
Robotics Test Engineer USD 100K-150KASan | Bitbucket Pipelines | C++ | CI/CD | DebuggingDirect hire | Mentorship | Ownership of test suitesMid-level Full TimeOnsite - Austin, TX20h ago
-
AI Feedback | Checkpointing | Cost Performance | Cost-performance tradeoffs | Data Decontamination401k matching | Country specific visa support | Flexible work arrangements | Medical, dental, and vision options | Parental leaveSenior-level Full TimePalo Alto, California, United States20h ago
-
AI Engineer 1 - Compliance, Quality & Testing USD 80K-150KAI-assisted testing | CI/CD | Code review | Cypress | Defect Tracking401k match | Dental insurance | Holidays | Medical insurance | Paid time offEntry-level Full TimeWashington, DC20h ago
-
Machine Learning Engineer, ML Flywheel USD 136K-173KC++ | Data Pipelines | Distributed Systems | ML Ops | Machine LearningAnnual bonus program | Company benefits program | Equity incentive plan | Hybrid workMid-level Full TimeMountain View, CA, USA21h ago
-
Software Engineer, Applied AI USD 190K-280KContext engineering | Data Processing | Data Storage | Debugging | Docker401k match | Dental insurance | Health insurance | Hybrid work model | Professional developmentSenior-level Full TimeNew York City R21h ago