Lead Data Engineer (Python, Databricks, SQL)
IL-70 W Madison St, 8th Fl, United States
USD 150K-170K Senior-level Full Time
Tasks
- Build ETL pipelines
- Define secure scalable data architecture
- Deliver end-to-end data engineering solutions
- Develop analytics data consumption
- Develop solutions using Databricks Python PySpark and SQL Server
- Drive innovation through proofs of concept
- Embed risk management and security privacy by design
- Ensure regulatory compliance
- Implement automation testing and CI CD
- Improve SDLC throughput via intake estimation and KPI management
- Modernize data platforms
- Optimize data ingestion transformation and reporting
- Oversee security and compliance adherence
- Set up data observability monitoring
- Translate business objectives into cloud data solutions
Perks/Benefits
- 401k
- Dental insurance
- Disability insurance
- Health savings account
- Holidays
- Hybrid work arrangement
- Life insurance
- Medical insurance
- Paid time off
- Parental leave
- Sick leave
- Vision insurance
Skills/Tech-stack
Automation | Azure | Azure Blob | Azure Blob Storage | Azure DevOps | Azure Functions | Azure SQL | Azure Storage | Blob Storage | CI/CD | Cognos | Data Ingestion | Data Optimization | Data Transformation | Data pipeline | Databricks | Databricks SQL | Databricks Unity | Databricks Unity Catalog | ETL | GitHub Copilot | Integration Services | Jira | Observability | Power BI | PySpark | Python | REST APIs | SQL | SQL Server | SQL Server Integration Services | SQL Server integration | SSIS | Scala | Server Integration Services | Tableau | Testing | Unity Catalog
Education
Bachelor of Engineering | Bachelor of Science | Master of Science | PhD
Roles
Data Engineer | Engineer | Lead | Lead Data | Lead Data Engineer
Related jobs
-
Lead Data Engineer USD 115K-130KAgile | Apache Airflow | Azure Data | Azure Data Factory | Cloud Orchestration401k | Dental insurance | Medical insurance | Paid leave | Tuition reimbursementSenior-level Full TimeUniversal City, CALIFORNIA, United States5h ago
-
Senior Engineer Embedded System Engineer USD 94K-125K8D methodology | A3 problem solving | Board Design | Bus Traces | Bus analyzerLife, accident, and disability insurance | Medical/Dental/Vision insurance | Paid sick leave | Paid vacation time | Tax-advantaged flexible spendingSenior-level Full TimeIrvine, CA, United States9h ago
-
Senior Systems Analyst – Robotic Algorithms and Control USD 134K-201KC# | C++ | CAD Tools | Classical control | Computer VisionSenior-level Full TimeSunnyvale, CA, United States10h ago
-
Senior Research Engineer, Voice + Speech USD 200K-400KData Pipelines | LLM | Language Processing | Machine Learning | Model EvaluationDaily meals snacks | Disability benefits | Fertility benefits | Life insurance | Medical/Dental/VisionSenior-level Full TimeNew York City10h ago
-
Staff Research Engineer, Voice + Speech USD 200K-400KConversational AI | Data Pipelines | Deep learning | Information Retrieval | LLM DeploymentDaily lunches and snacks | Disability benefits | Fertility and family building benefits | Life insurance | Medical/Dental/Vision insuranceSenior-level Full TimeNew York City10h ago
-
Senior Research Engineer, Voice + Speech USD 200K-400KData Pipelines | Deep learning | Information Retrieval | LLM Deployment | Language ModelsDaily meals | Dental insurance | Disability insurance | Health insurance | Life insuranceSenior-level Full TimeSan Francisco11h ago
-
API Gateway | AWS Glue | Access Control | Apache Airflow | Apache AtlasSenior-level Full TimeFAIRFAX, VA, United States11h ago
-
Senior Enterprise Data Operations Support Engineer (Tier-3 Expert Engineering Support) USD 125K-153KAWS GovCloud | Apache Airflow | Apache Kafka | Apache Spark | Azure GovernmentSenior-level Full TimeFAIRFAX, VA, United States11h ago
-
AI Developer - Model Creation & Full Stack USD 150K-175KAWS | Angular | Azure | CI/CD | D3.jsRemote work | USPS Public Trust Clearance eligibleMid-level Full TimeWork from home, VA, United States R11h ago
-
Enterprise Data Engineering & Operations Lead SME USD 107K-190KAWS GovCloud | Access Control | Apache Kafka | Apache Spark | Azure GovernmentSenior-level Full TimeFAIRFAX, VA, United States11h ago
-
Software/Embedded Systems Engineer USD 135K-158KAgile Development | C++ | Change Control | DoD Systems | Embedded SoftwareTravel 25%Senior-level Full TimeArlington, VA, United States11h ago
-
Data/ML Scientist SME USD 105K-150KAWS GovCloud | Anomaly Detection | Apache Spark | Bayesian Causal Inference | Bayesian MethodsMid-level Full TimeFAIRFAX, VA, United States11h ago
-
AI Data Engineer USD 165K-225KAudit Columns | CI/CD | Cortex Analyst | Cortex Complete | DBT401k company match | Flexible paid time off | Learning and development | Medical benefits | Paid parental leaveSenior-level Full TimePhiladelphia, PA, United States11h ago
-
Lead AI Engineer USD 180K-280KCI/CD | ColBERT | Docker | Faiss | Fine Tuning401k match | Bonus | Childcare benefits | Dental insurance | Disability insuranceSenior-level Full TimeQuincy, MA, United States12h ago
-
API Integration | AWS | AWS Glue | Batch Processing | Code reviewSenior-level Full TimeIndianapolis, IN, United States R13h ago
-
Senior-level Full TimeUnited States13h ago
-
Senior-level Full TimeUnited States13h ago
-
Software engineer, generative AI USD 119K-292KAWS | Agentic Workflows | Asyncio | Azure | Docker401k | Cancer testing support | Company holidays | Company off-sites | Company stock optionsMid-level Full TimeSan Francisco, CA13h ago
-
AI engineer USD 152K-315KAWS | Azure | Cloud Computing | Deep learning | GCP401k | Company offsites | Dental insurance | Fertility support | Flexible spending accountMid-level Full TimeSan Francisco, CA13h ago
-
Computational Biologist - Protein Engineering USD 150K-250KAWS | Amazon Web Services | CUDA | Conda | Deep learningRelocation supportEntry-level Full TimeSan Francisco, CA, US14h ago
-
Entry-level Full TimeSan Francisco, CA, US14h ago
-
Applied Math Libraries Engineer (SPU) USD 120K-170KArithmetic intensity | Atomics | BLAS | Bandwidth) | C#Entry-level Full TimeRedwood City14h ago
-
Kernel Engineer — Scientific Computing (SPU) USD 120K-170KAtomic operations | C++ | CUDA | Concurrency | DebuggingEntry-level Full TimeRedwood City14h ago
-
Product Engineer USD 120K-200KAWS | FFmpeg | Next.js | Node.js | PostgreSQL401k access | Flexible PTO | Health, dental, vision coverage | Paid mealsEntry-level Full TimeSan Francisco, CA, US14h ago
-
ADA | API Development | Application Programming | Application Programming Interface | C#Active secret clearanceSenior-level Full TimeAtlanta, Georgia15h ago