Lead Data Engineer (Python, Databricks, SQL)
IL-70 W Madison St, 8th Fl, United States
USD 150K-170K Senior-level Full Time
Tasks
- Build ETL pipelines
- Define secure scalable data architecture
- Deliver end-to-end data engineering solutions
- Develop analytics data consumption
- Develop solutions using Databricks Python PySpark and SQL Server
- Drive innovation through proofs of concept
- Embed risk management and security privacy by design
- Ensure regulatory compliance
- Implement automation testing and CI CD
- Improve SDLC throughput via intake estimation and KPI management
- Modernize data platforms
- Optimize data ingestion transformation and reporting
- Oversee security and compliance adherence
- Set up data observability monitoring
- Translate business objectives into cloud data solutions
Perks/Benefits
- 401k
- Dental insurance
- Disability insurance
- Health savings account
- Holidays
- Hybrid work arrangement
- Life insurance
- Medical insurance
- Paid time off
- Parental leave
- Sick leave
- Vision insurance
Skills/Tech-stack
Automation | Azure | Azure Blob | Azure Blob Storage | Azure DevOps | Azure Functions | Azure SQL | Azure Storage | Blob Storage | CI/CD | Cognos | Data Ingestion | Data Optimization | Data Transformation | Data pipeline | Databricks | Databricks SQL | Databricks Unity | Databricks Unity Catalog | ETL | GitHub Copilot | Integration Services | Jira | Observability | Power BI | PySpark | Python | REST APIs | SQL | SQL Server | SQL Server Integration Services | SQL Server integration | SSIS | Scala | Server Integration Services | Tableau | Testing | Unity Catalog
Education
Bachelor of Engineering | Bachelor of Science | Master of Science | PhD
Roles
Data Engineer | Engineer | Lead | Lead Data | Lead Data Engineer
Related jobs
-
Data Engineer USD 100K-128KData Governance | Data Modeling | Data Security | Databricks | ELT401k match | Adoption Assistance | Community volunteer opportunities | Continuing education support | Dental insuranceMid-level Full TimeSouth Sioux City, NE, United States R10h ago
-
Cognos Analytics | Control-M | DB2 | Database Administration | ETLRemote workSenior-level ContractLincoln, United States R12h ago
-
Senior Data Engineer TS/SCI Clearance USD 160K-220KAWS | Cloud Native | Data Visualization | Database Design | Database performanceBest place to work recognition | Employee development | Full employee approach | High employee morale and retentionSenior-level Full TimeHuntsville, United States12h ago
-
Data Engineer USD 148K-263KAPI | Apache Kafka | Apache Spark | Cassandra | Distributed SystemsDisability insurance | Health insurance | Holiday pay | Learning and development | Life insuranceMid-level Full TimeUSA-Remote Work R12h ago
-
A/B | A/B Testing | B testing | CI/CD | Collaborative FilteringSenior-level Full TimeSan Jose, California, United States12h ago
-
Data cleaning | Data collection | Deep learning | Machine Learning | Model EvaluationSenior-level Full TimeSan Jose, California, United States12h ago
-
Software Engineer - AI Agent Memory Infrastructure USD 156K-316KConflict Resolution | Data Compression | Data Ingestion | Data Modeling | Data StorageMid-level Full TimeSan Jose, California, United States12h ago
-
GenAI Engineer USD 93K-163KAWS Bedrock | Agentic Workflows | C++ | CI/CD | CohereHealth and wellness benefits | Mentorship | Professional developmentEntry-level Full TimeArlington/Rosslyn, Virginia, United States13h ago
-
Senior GenAI Engineer USD 102K-171KAPI Development | AWS Bedrock | Agentic Workflows | CI/CD | CohereSenior-level Full TimeArlington/Rosslyn, Virginia, United States13h ago
-
Data Scientist - Platform Infrastructure USD 127K-189KData Governance | Data Modeling | Data Pipelines | Data Quality | ETLMid-level Full TimeLos Angeles, California, United States13h ago
-
C++ | Data Compression | Data Ingestion | Data Processing | Data StorageSenior-level Full TimeSan Jose, California, United States13h ago
-
Computer Vision | Data Pipelines | Language Models | Language Processing | Large Language ModelsSenior-level Full TimeBellevue, WA | Menlo Park, CA14h ago
-
Senior Software Engineer, Cloud Databases USD 174K-252KAnalytical processing | Benchmarking | C++ | Cloud Databases | Cloud platformSenior-level Full TimeKirkland, WA, USA14h ago
-
SOC Architect, XProf USD 147K-211KC# | C++ | Compiler profiling | Data Analysis | Data VisualizationSenior-level Full TimeSunnyvale, CA, USA14h ago
-
Principal Engineer, Data Protection USD 307K-427KArtificial Intelligence | Data Protection | Data Transformation | Data labeling | Machine LearningSenior-level Full TimeSunnyvale, CA, USA; New York, NY, …14h ago
-
Data Engineer, Global Business and Operations USD 130K-187KBigQuery | Data Governance | Data Marts | Data Modeling | Data PipelinesMid-level Full TimeNew York, NY, USA14h ago
-
Data Engineer USD 110K-110KAutomated testing | Data Architecture | Data Modeling | Data Security | Data VisualizationMid-level Full TimeAnnapolis, MD, US16h ago
-
Sr. Machine Learning Engineer USD 91K-177KAlgorithms | Anomaly Detection | Apache Airflow | Data Analysis | Deep learning401k plan | Employee recognition | Employee stock purchase plan | Health insurance | Paid time offSenior-level Full TimeIrvine, CA, US16h ago
-
Staff Software Engineer - Core Ingest USD 191K-224KAgile Development | Apache Kafka | Distributed Systems | Docker | Fault ToleranceHealth insurance | Paid time off | Remote work optionsSenior-level Full TimeUnited States, Remote R21h ago
-
Staff Software Engineer - Data Query USD 191K-224KAgile | Automated testing | Big Data | C++ | Data StructuresSenior-level Full TimeUnited States, Remote R21h ago
-
ArcGIS Pro | Arcpy | Bokeh | Dash | GDAL401k | Dental insurance | Health insurance | Vision insuranceSenior-level Full TimeFayetteville, North Carolina, United States1d ago
-
ArcGIS Pro | Arcpy | Bokeh | Dash | GDAL401k | Dental insurance | Health insurance | TS/SCI clearance | Vision insuranceSenior-level Full TimeSneads Ferry, North Carolina, United States1d ago
-
Data Engineer - Mid-Level USD 130K-160KAirflow | Automated Deployment | Automated testing | CI/CD | Control workflows401k matching | Dental insurance | Health insurance | Lunch and snacks provided | Maternity & paternity leaveMid-level Full TimeEl Segundo, California, United States1d ago
-
Software Engineer – Surgical Robot Manufacturing USD 127K-192KAutomated testing | Control Systems | HTML | JSON | JavaScriptMid-level Full TimeSunnyvale, CA, United States1d ago
-
Senior-level Full TimeRemote - United States R1d ago