Lead Data Engineer
Chicago - Illinois - Wacker, United States
USD 129K-161K Senior-level Full Time
Tasks
- Build data ingestion pipelines into Enterprise Data Hub
- Decommission legacy data infrastructure
- Develop Databricks notebooks for data processing
- Develop data transformation pipelines
- Document data flows and dependencies
- Drive automation and process optimization
- Implement CI/CD pipelines
- Implement ETL ELT migration processes
- Implement batch and real-time data integration
- Implement data cataloging and classification
- Implement data quality checks and monitoring
- Integrate data governance and lineage
- Manage change data capture processes
- Mentor data engineers
- Optimize data storage for query performance and cost
- Support BI analytics and AI data needs
- Troubleshoot pipeline failures and performance issues
Perks/Benefits
Skills/Tech-stack
Automation | Azure Data | Azure Data Factory | Azure Purview | BI reporting | Batch Processing | CI/CD | Change Data Capture | Data Capture | Data Factory | Data Governance | Data Management | Data Modeling | Data Pipelines | Data Warehousing | Databricks | Delta | Delta Lake | ELT | ETL | Integration Services | Master Data | Master Data Management | Parquet | Profisee | Real Time | Real-time Streaming | SQL Server | SQL Server Integration Services | SQL Server Management | SQL Server integration | Scripting | Server Integration Services | Server management | Tableau
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Roles
Data Engineer | Engineer | Lead | Lead Data | Lead Data Engineer
Related jobs
-
Data Engineer (remote) USD 85K-100KAgile | Apache Hive | Apache Impala | Apache Spark | Artificial Intelligence401k match | Employee assistance program | Flexible schedule | Health insurance | Life insuranceMid-level Full TimeWork From Home, United States R9h ago
-
AWS | Alerting | CI/CD | Compliance | Data LakeCareer growth | Dental insurance | Medical insurance | Paid leave | Retirement plansExecutive-level Full TimeHouston, TX, United States10h ago
-
Senior Data Engineer Principal & Lead –on Site USD 165K-210KAWS Lambda | Amazon Kinesis | Amazon Redshift | Amazon S3 | Data ModelingDay 1 impact | On-site role | Own product and infrastructure | Shape technical vision | Small senior teamSenior-level Full TimeSan Francisco, United States13h ago
-
SOFTWARE ENGINEER - Apache Spark/Big Data - 7+ yrs of Experience - TS/SCI w/Poly clearance is required - ID A USD 234K-241KApache Airflow | Apache Spark | Bash | CSV | Configuration Management401k retirement plan | Dental insurance | Disability coverage | Federal Holidays | Health insuranceMid-level Full TimeFort George G Meade, United States13h ago
-
Software Engineer, Backend and Infrastructure USD 156K-316KAlerting | CI/CD | Database Design | Distributed Systems | HDFSMid-level Full TimeSan Jose, California, United States13h ago
-
Senior Software Engineer, Backend and Infrastructure USD 156K-387KAlerting | CI/CD | Cloud IDE | Database | HDFSSenior-level Full TimeSan Jose, California, United States13h ago
-
Benchmarking | Data Pipelines | Evaluation metrics | Interpretability | LLM EvaluationSenior-level Full TimeSan Jose, California, United States13h ago
-
Robotics Engineer - Autonomous Drone System USD 167K-230KAnalytics | Anomaly Detection | Autonomous Navigation | BVLOS | C++Senior-level Full TimeFremont, CA14h ago
-
Robotics Firmware & Electrical Engineer USD 170K-240KC# | C++ | CAN | Electrical layout | Embedded SystemsSenior-level Full TimeFremont, CA14h ago
-
Agent Orchestration | Benchmarking | Data Generation | Data Pipelines | EvaluationSenior-level Full TimeBellevue, WA | Menlo Park, CA …14h ago
-
Abuse prevention | Adversarial Machine Learning | Agentic Models | Benchmarks | Code reviewMid-level Full TimeMenlo Park, CA14h ago
-
AI Agents | AppScript | Artificial Intelligence | Business Intelligence | Change ManagementSenior-level Full TimeSan Bruno, CA, USA14h ago
-
Staff Software Engineer, Biglakes, BigQuery USD 207K-300KApache Arrow | Apache Flink | Apache Spark | Big Data | Big data analyticsSenior-level Full TimeSunnyvale, CA, USA; Kirkland, WA, USA14h ago
-
AI Engineer USD 120K-173KArtificial Intelligence | CI/CD | Data Preprocessing | Feature Engineering | MLOpsEmployee resource groups | Flexible family care | Hybrid work | Medical dental prescription coverage | Paid HolidaysSenior-level Full TimeDearborn, MI, United States23h ago
-
Senior Software Engineer, AI Training & Infrastructure USD 200K-300KAWS | Alerting | Algorithms | Automated testing | AzureSenior-level Full TimeSan Mateo1d ago
-
Autonomy | Closed Loop | Closed loop control | Computer Vision | Control SystemsSenior-level Full TimeNew York, New York, USA1d ago
-
Principal, Data Engineer USD 107K-216KAzure | Azure Data | Azure Data Factory | Azure Purview | Azure SynapseSenior-level Full Time11 Keewaydin Dr, Salem NH, United …1d ago
-
Senior Data & AI Engineer USD 139K-230KAWS | Agent Orchestration | Agent systems | Apache Spark | Artificial Intelligence401k match | Health insurance | Mental health counseling | Paid time off | Volunteer encouragementSenior-level Full TimeHartford - Tower, United States1d ago
-
AI Software Engineer - New College Graduate USD 120K-134KAI machine learning | Automation | Confluence | Data Engineering | Go401k retirement plan | Disability insurance | Health and wellness benefits | Life insurance | Paid HolidaysEntry-level Full TimeCohesity - Santa Clara, United States1d ago
-
Senior-level Full TimePortland, Oregon, United States1d ago
-
Data Engineer II (Snowflake, AWS, DBT) USD 126K-208KAmazon Web Services | DBT | Data Cleansing | Data Governance | Data Mapping401k match | Health insurance | Mental health counseling | Paid Holidays | Paid time offMid-level Full TimeHartford - Tower, United States1d ago
-
Data Engineer II USD 110K-110KAlerting | Cloud Platforms | Data Modeling | Data Pipelines | Data QualityEmployee benefit allowance | Paid Holidays | Paid sick leave | Paid vacationMid-level Full TimeOklahoma City - 3115 N Lincoln …1d ago
-
Senior Project Engineer, Embedded Software USD 93K-165KAC motor | AC motor control | ADC | ARM | ASMDisability benefits | Discount programs | Employee stock purchase plan | Life insurance | Medical/Dental/Vision insuranceSenior-level Full TimeTowson MD USA - 701 E …1d ago
-
Dashboarding | Data Analysis | Data Modeling | Data Querying | Data VisualizationCareer development opportunities | Education assistance program | Flexible work options | Generous leave accruals | Health benefitsSenior-level Full TimeAlvarado Square, United States R1d ago
-
ARM | C# | C++ | Continuous integration | DebuggingCompany-Paid Holidays | Disability insurance | Health insurance | Life insurance | Paid time offMid-level Full TimeAZCH02, United States1d ago