Lead Data Engineer
Chicago - Illinois - Wacker, United States
USD 129K-161K Senior-level Full Time
Tasks
- Automate data engineering workflows
- Build data ingestion pipelines
- Create Tableau dashboards for analytics
- Develop Databricks notebooks
- Develop data transformation pipelines
- Document data flows and dependencies
- Implement CI/CD pipelines
- Implement batch and real-time data integration
- Implement data cataloging classification and lineage
- Implement data quality checks
- Implement master data management
- Maintain ETL/ELT processes
- Maintain data governance policies and compliance
- Manage data storage for query performance
- Mentor data engineers
- Migrate legacy data to Enterprise Data Hub
- Monitor data pipeline health
- Optimize pipeline performance
- Redirect pipelines during decommissioning
- Set up Change Data Capture processes
- Troubleshoot pipeline failures
Perks/Benefits
Skills/Tech-stack
Automation | Azure | Azure Data | Azure Data Factory | Azure Purview | Big Data | Big data processing | Business Intelligence | CI/CD | Change Data Capture | Data Capture | Data Factory | Data Governance | Data Management | Data Modeling | Data Processing | Data Quality | Data Warehousing | Databricks | Databricks Notebooks | Delta | ELT | ETL | Integration Services | Machine Learning | Master Data | Master Data Management | Parquet | Predictive Analytics | Profisee | SQL Server | SQL Server Integration Services | SQL Server Management | SQL Server integration | Scripting | Server Integration Services | Server management | Tableau
Education
Bachelor of Arts | Bachelor of Engineering | Bachelor of Science | Master of Science
Roles
Related jobs
-
Data Engineer (remote) USD 85K-100KAgile | Apache Hive | Apache Impala | Apache Spark | Artificial Intelligence401k match | Employee assistance program | Flexible schedule | Health insurance | Life insuranceMid-level Full TimeWork From Home, United States R8h ago
-
AWS | Alerting | CI/CD | Compliance | Data LakeCareer growth | Dental insurance | Medical insurance | Paid leave | Retirement plansExecutive-level Full TimeHouston, TX, United States10h ago
-
Senior Data Engineer Principal & Lead –on Site USD 165K-210KAWS Lambda | Amazon Kinesis | Amazon Redshift | Amazon S3 | Data ModelingDay 1 impact | On-site role | Own product and infrastructure | Shape technical vision | Small senior teamSenior-level Full TimeSan Francisco, United States12h ago
-
SOFTWARE ENGINEER - Apache Spark/Big Data - 7+ yrs of Experience - TS/SCI w/Poly clearance is required - ID A USD 234K-241KApache Airflow | Apache Spark | Bash | CSV | Configuration Management401k retirement plan | Dental insurance | Disability coverage | Federal Holidays | Health insuranceMid-level Full TimeFort George G Meade, United States12h ago
-
Algorithms | C++ | Computational lithography | Data Structures | Image ProcessingFinancial benefits | Health benefits | Wellness programsSenior-level Full TimeCalifornia, Sunnyvale12h ago
-
Software Engineer, Backend and Infrastructure USD 156K-316KAlerting | CI/CD | Database Design | Distributed Systems | HDFSMid-level Full TimeSan Jose, California, United States12h ago
-
Senior Software Engineer, Backend and Infrastructure USD 156K-387KAlerting | CI/CD | Cloud IDE | Database | HDFSSenior-level Full TimeSan Jose, California, United States12h ago
-
Benchmarking | Data Pipelines | Evaluation metrics | Interpretability | LLM EvaluationSenior-level Full TimeSan Jose, California, United States13h ago
-
Computer Vision | Data Processing | Deep learning | Feature Engineering | Language ModelsSenior-level Full TimeSan Jose, California, United States13h ago
-
Robotics Engineer - Autonomous Drone System USD 167K-230KAnalytics | Anomaly Detection | Autonomous Navigation | BVLOS | C++Senior-level Full TimeFremont, CA13h ago
-
Robotics Control Engineer - Manipulation USD 162K-240KABB Rapid | C++ | Calibration | Cause analysis | CommissioningSenior-level Full TimeMenlo Park, CA | Fremont, CA13h ago
-
Agent Orchestration | Benchmarking | Data Generation | Data Pipelines | EvaluationSenior-level Full TimeBellevue, WA | Menlo Park, CA …13h ago
-
Abuse prevention | Adversarial Machine Learning | Agentic Models | Benchmarks | Code reviewMid-level Full TimeMenlo Park, CA13h ago
-
Senior Research Software Engineer, Labs, AI/ML USD 174K-252KC++ | Data Structures | Data Structures and Algorithms | Deep learning | Distributed ComputingSenior-level Full TimeMountain View, CA, USA; Zürich, Switzerland14h ago
-
Software Engineer III, AI/ML, Google Cloud AI USD 147K-211KC++ | Data Processing | Debugging | GenAI | Google CloudSenior-level Full TimeSunnyvale, CA, USA14h ago
-
AI Agents | AppScript | Artificial Intelligence | Business Intelligence | Change ManagementSenior-level Full TimeSan Bruno, CA, USA14h ago
-
Staff Software Engineer, On-Device Machine Learning USD 207K-300KAndroid | Data Processing | Data Structures | Data Structures and Algorithms | DebuggingSenior-level Full TimeSunnyvale, CA, USA14h ago
-
Senior Engineering Analyst, AI, Safety USD 159K-231KAbuse detection | C++ | Data Analysis | Fine Tuning | Fraud DetectionEquity compensation | Health insurance | Mentorship | Paid time off | Retirement benefitsSenior-level Full TimeKirkland, WA, USA; Seattle, WA, USA14h ago
-
Staff Software Engineer, Biglakes, BigQuery USD 207K-300KApache Arrow | Apache Flink | Apache Spark | Big Data | Big data analyticsSenior-level Full TimeSunnyvale, CA, USA; Kirkland, WA, USA14h ago
-
Senior Software Engineer, ML Performance Infrastructure USD 174K-252KC# | C++ | Compute technology | Data Storage | Data StructuresSenior-level Full TimeSunnyvale, CA, USA14h ago
-
Software Engineer, Next Generation AI/ML Infrastructure USD 147K-211KC++ | Data Processing | Data Storage | Distributed Processing | Feature StoresMid-level Full TimeSunnyvale, CA, USA14h ago
-
Senior Software Engineer, Jules AI, Labs, AI/ML USD 174K-252KAI infrastructure | Agent systems | Context engineering | Distributed Computing | Generative AISenior-level Full TimeMountain View, CA, USA; New York, …14h ago
-
Senior Software Engineer, Connect Sales, CRM, GenAI, Ads USD 174K-252KComputer Vision | Data Processing | Debugging | Distributed Computing | Generative AISenior-level Full TimeMountain View, CA, USA14h ago
-
AI Engineer USD 120K-173KArtificial Intelligence | CI/CD | Data Preprocessing | Feature Engineering | MLOpsEmployee resource groups | Flexible family care | Hybrid work | Medical dental prescription coverage | Paid HolidaysSenior-level Full TimeDearborn, MI, United States22h ago
-
Senior Software Engineer, AI Training & Infrastructure USD 200K-300KAWS | Alerting | Algorithms | Automated testing | AzureSenior-level Full TimeSan Mateo23h ago