Lead Data Engineer (Python, Databricks, SQL)
IL-Chicago, 70 W Madison St Fl 8, United States
USD 150K-165K Senior-level Full Time
Tasks
- Build analytics and data consumption solutions
- Build solutions with Databricks Python and PySpark
- Design secure scalable cloud data architecture
- Develop ETL pipelines
- Develop and optimize SQL Server pipelines
- Drive modernization with proofs of concept
- Ensure regulatory compliance with audit partnership
- Implement automated testing and CI/CD
- Improve SDLC throughput
- Manage intake estimation and KPIs
- Oversee risk management and security by design
- Own end to end data strategy delivery
- Provide technical leadership and mentorship
- Set up observability and continuous improvement
- Translate business objectives into cloud data solutions
Perks/Benefits
- 401k
- Disability insurance
- Health savings account
- Holidays
- Hybrid work schedule
- Life insurance
- Medical, dental & vision coverage
- Paid sick leave
- Paid time off
- Parental leave
- Recognition program
- Wellbeing support
Skills/Tech-stack
Automation | Azure | Azure Databricks | Azure DevOps | Azure SQL | CI/CD | Cognos | Databricks | Databricks SQL | Databricks Unity | Databricks Unity Catalog | DevOps | ETL | GitHub Copilot | Integration | Jira | Observability | Power BI | PySpark | Python | REST APIs | Release Management | SDLC | SQL Server | Tableau | Testing | Unity Catalog
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Roles
Data Engineer | Engineer | Lead | Lead Data | Lead Data Engineer
Related jobs
-
Software Engineer, Data (L1) USD 75K-130KAI Platform | APIs | Apache Airflow | Apache Iceberg | Apache Spark401k match | Calm subscription | Career growth and learning opportunities | Commuter benefits options | Dental insuranceMid-level Full TimeAustin, TX8h ago
-
Solutions Engineer – Robotics & Autonomous Driving USD 152K-500K2D Imaging | 3D Imaging | Autonomous Driving | C++ | CameraCommuter stipend | Generous PTO | Global travel | Health, dental, vision coverage | Learning and development stipendSenior-level Full TimeSan Francisco, CA; New York, NY10h ago
-
AI Developer and Data Lead USD 91K-184K.NET | AI Security | API Development | AWS Lambda | Amazon AuroraSenior-level Full TimeWashington, DC12h ago
-
Machine Learning Research Engineer USD 161K-189KAWS | Azure | Bias Variance | Bias-Variance Tradeoff | C++Flexible hybrid work model | Mental health counseling | Mentorship programs | Paid parental leave | Paid volunteer time offMid-level Full TimeNew York, US, New York12h ago
-
Big Data | Cloud Cost Optimization | Cost Optimization | Data Governance | Data Security401k plan | Dental insurance | Health insurance | Life insurance | Paid time offMid-level Full TimeSpringdale, AR Terminal Office - Springdale, …13h ago
-
Senior Data Engineer - Logistics USD 38K-38KBig Data | Cloud Cost Optimization | Column-Level Security | Cost Optimization | Data Governance401k plan | Dental insurance | Health insurance | Life insurance | Paid time offSenior-level Full TimeSpringdale, AR Terminal Office - Springdale, …13h ago
-
Sr. Application Software Engineer, Data Analytics USD 160K-225KAngular | C# | CI/CD | Computer Vision | Continuous integrationExtended hours | Travel | Weekend workSenior-level Full TimeBastrop, TX13h ago
-
AI Engineer - Automation/Robotics USD 104K-110KAlgorithms | Azure Machine Learning | Computer Vision | Data Structures | DebuggingOnsite workEntry-level Full TimeRedmond, WA14h ago
-
Lead Agentic AI Developer USD 172K-186KAI Development | AI development lifecycle | Agentic AI | Amazon Bedrock | Artificial IntelligenceSenior-level Full TimeSeattle, WA, United States14h ago
-
Mid-level Full TimeEl Segundo15h ago
-
Senior-level Full TimeSan Francisco, California15h ago
-
Foundation Model Data, Software Engineer USD 213K-263KApache Beam | Apache Flume | C++ | CI/CD | Distributed Systems401k match | Baby bonding leave | Dental insurance | Disability insurance | Health insuranceSenior-level Full TimeMountain View, CA, USA; San Francisco, …15h ago
-
Senior-level Full TimeOnsite - Austin, TX15h ago
-
Data parallelism | Diffusion Models | Efficient Attention | Expert parallelism | FlaxSenior-level Full TimeMountain View, California, United States, New …15h ago
-
AWS Bedrock | Agent systems | Anthropic API | Autogen | Azure401k matching program | Adoption Assistance | Development and career growth opportunities | Fertility treatments | Flexible work schedulesSenior-level Contract Full TimeRemote, OR, United States R16h ago
-
Senior Data Engineer USD 90K-110KAWS | Agile | Apache NiFi | Data Architecture | Data ModelingAutonomy | Flexible working hours | Global employee assistance programme | Online training videos | Teambuilding eventsSenior-level Full TimeNew York, United States16h ago
-
Customer Engineer, AI Solutions USD 110K-140KAI | API | Customer adoption | Integrations | Onboarding401k match | Dental insurance | Equity | Health insurance | Hybrid workMid-level Full TimeWashington, District of Columbia, USA16h ago
-
Customer Engineer, AI Solutions USD 110K-140KCustomer adoption | Enterprise Software | Integrations | Product Roadmap | Product demonstrations401k match | Dental insurance | Health insurance | Hybrid work | Vision insuranceMid-level Full TimeNew York, New York, USA16h ago
-
Data Platform Engineer USD 85K-120KAlembic | App Service | Azure | Azure App | Azure App Service401k match | Annual discretionary bonus program | Bereavement leave | Commuter spending accounts | Dental insuranceMid-level Full TimeChicago, IL (Downtown)16h ago
-
Robotics Test Engineer USD 100K-150KASan | Bitbucket Pipelines | C++ | CI/CD | DebuggingDirect hire | Mentorship | Ownership of test suitesMid-level Full TimeOnsite - Austin, TX17h ago
-
AI Engineer 1 - Compliance, Quality & Testing USD 80K-150KAI-assisted testing | CI/CD | Code review | Cypress | Defect Tracking401k match | Dental insurance | Holidays | Medical insurance | Paid time offEntry-level Full TimeWashington, DC17h ago
-
Machine Learning Engineer, ML Flywheel USD 136K-173KC++ | Data Pipelines | Distributed Systems | ML Ops | Machine LearningAnnual bonus program | Company benefits program | Equity incentive plan | Hybrid workMid-level Full TimeMountain View, CA, USA17h ago
-
Software Engineer, Applied AI USD 190K-280KContext engineering | Data Processing | Data Storage | Debugging | Docker401k match | Dental insurance | Health insurance | Hybrid work model | Professional developmentSenior-level Full TimeNew York City R17h ago
-
Data Engineer USD 74K-133KAgile | Apache Airflow | BigQuery | Cloud Composer | Cloud Data401k retirement plan | Dental insurance | Disability insurance | Flexible time off | Health insuranceMid-level Full TimeLisle, IL, United States R17h ago
-
Lead Data Scientist USD 185K-215KA/B | A/B Testing | B testing | Credit Risk | Credit risk modeling401k plan | Cell phone reimbursement | Dental insurance | Flexible PTO | Gym membership discountSenior-level Full TimeNew York City (Hybrid) R18h ago