Data Engineer
Tasks
- Build and tune Apache Spark jobs for large scale processing
- Build entity resolution and record linkage logic
- Collaborate on semantic models and datasets for dashboards
- Design build and operate batch and real time data pipelines
- Establish and enforce data governance with Unity Catalog
- Implement CI CD for data and infrastructure
- Implement automated data quality validation profiling and anomaly detection
- Implement medallion architecture with Delta Lake and Delta Live Tables
- Ingest and process high volume IoT and historian data
- Model analytics ready datasets and serving layers
- Provide governed feature ready data for ML workflows
- Set up monitoring logging and observability for pipelines
- Support production incidents and create runbooks
Perks/Benefits
- 401k match
- Dental insurance
- Flexible paid time off
- Health insurance
- Life insurance
- Long-term disability
- Paid parental leave
- Short-term disability
- Vision insurance
Skills/Tech-stack
Airflow | Apache Spark | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DevOps | CI/CD | DBT | Data Factory | Data Governance | Data Lake Storage | Data Modeling | Data Quality | Databricks | Delta Lake | Delta Live Tables | Delta Live) | Docker | Entity Resolution | Event Hubs | GitHub Actions | Infrastructure as Code | Kafka | Logging | Monitoring | Observability | Power BI | PySpark | Python | Record Linkage | SQL | Snowflake | Spark SQL | Spotfire | Structured Streaming | Terraform | Unity Catalog | “as-code”
Roles
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R25d ago
-
Mid-level Full TimeTel Aviv-Yafo, Tel Aviv District, IL5h ago
-
Lead Data Engineer USD 110K-120KAmazon Redshift | Amazon Web Services | Azure | Azure Synapse | CDCSenior-level Full TimeWarsaw, IN, US7h ago
-
Business Intelligence | Dashboard Development | Data Architecture | Data Modeling | Data PipelinesSenior-level ContractLansing, United States7h ago
-
Software Infrastructure Developer (Agentic AI Focus) USD 160K-213KAWS | Agent SDK | Anthropic API | Asynchronous | AzureSenior-level Full TimeTel Aviv-Yafo, Tel Aviv District, IL7h ago
-
Embedded System Engineer Co-Op USD 56K-60KC# | C++ | Communication Protocols | Electronic Hardware | Electronics designDental insurance | Disability insurance | Educational assistance | Flexible spending account | Health savings accountEntry-level Full TimeNorwich, New York8h ago
-
(General Hire) Machine Learning Engineer Graduate (TikTok Recommendation) - 2027 Start USD 116K-177KAlgorithms | C++ | Computer Vision | Data Structures | Deep learningEntry-level Full TimeSan Jose, California, United States8h ago
-
(General Hire) Machine Learning Scientist Graduate (TikTok Recommendation) - 2027 Start (PhD) USD 119K-177KAlgorithms | C plus plus | C# | Data Structures | Deep learningEntry-level Full TimeSan Jose, California, United States8h ago
-
Algorithms | C++ | Data Mining | Data Pipelines | Data StructuresEntry-level InternshipSan Jose, California, United States8h ago
-
C++ | Computer Vision | Data Mining | Data Pipelines | DatabasesEntry-level Full TimeSan Jose, California, United States8h ago
-
Computer Vision | Data-Driven Strategies | Data-driven | Language Models | Language ProcessingEntry-level Full TimeSeattle, Washington, United States8h ago
-
Machine Learning Engineer Intern (TikTok-Data-Search-Basic Ranking) - 2027 Start (PhD) USD 136K-205KAlgorithms | Data Pipelines | Data Structures | Databases | Distributed SystemsEntry-level InternshipSan Jose, California, United States8h ago
-
Backend Development | C++ | Computer Vision | Data Mining | Data PipelinesEntry-level Full TimeSan Jose, California, United States8h ago
-
Algorithms | Backend Services | C++ | Data Mining | Data PipelinesEntry-level InternshipSan Jose, California, United States8h ago
-
Backend systems | C++ | Computer Vision | Data Mining | Data PipelinesEntry-level Full TimeSan Jose, California, United States8h ago
-
Backend Development | C++ | Computer Vision | Data Mining | Data PipelinesEntry-level Full TimeSan Jose, California, United States8h ago
-
Research Engineer Graduate (Monetization Technology - Business Integrity) - 2027 Start USD 145K-250KAgentic Systems | Computer Vision | Data Pipelines | Debugging | DeepSpeedEntry-level Full TimeSan Jose, California, United States8h ago
-
Backend Development | C++ | Computer Vision | Data Mining | Data PipelinesEntry-level Full TimeSeattle, Washington, United States8h ago
-
Backend Development | C++ | Computer Vision | Data Mining | Data PipelinesEntry-level Full TimeSeattle, Washington, United States8h ago
-
Machine Learning Engineer Graduate (Monetization Technology - Ads Core Global) - 2027 Start USD 113K-250KAlgorithms | C++ | Data Structures | Deep learning | Diffusion ModelsEntry-level Full TimeSan Jose, California, United States8h ago
-
A/B | A/B Testing | Apache Flink | Apache Hive | Apache SparkEntry-level Full TimeSan Jose, California, United States8h ago
-
Advertising Technology | Backend Development | C++ | Data Analysis | DebuggingEntry-level Full TimeSan Jose, California, United States8h ago
-
Machine Learning Engineer Graduate (Monetization Technology - Ads Core Global) - 2027 Start (PhD) USD 113K-250KAdvertising | Algorithms | C plus plus | C# | Data StructuresEntry-level Full TimeSan Jose, California, United States8h ago
-
ML Infra Engineer Graduate (Ads Infra) - 2027 Start USD 113K-250KAI Agents | C plus plus | Cloud Native | Data Pipelines | Data ProcessingEntry-level Full TimeSan Jose, California, United States8h ago
-
Algorithms | C++ | Data Structures | Debugging | Deep learningEntry-level Full TimeSan Jose, California, United States8h ago