Data Engineer - Databricks
Chicago, IL, United States
USD 172K-237K (estimate) Mid-level Full Time
Tasks
- Build CI/CD pipelines for Databricks
- Build Delta Live Tables pipelines
- Configure Unity Catalog for governance
- Design ETL/ELT pipelines
- Document solutions and runbooks
- Implement medallion architecture
- Ingest streaming and incremental data
- Optimize lakehouse data platforms
- Orchestrate and monitor production workloads
- Prepare ML ready datasets with MLflow
- Query and profile datasets
- Set up PII masking
- Tune clusters for performance and cost
Perks/Benefits
- N/A
Skills/Tech-stack
ACID transactions | Airflow | Auto Loader | Azure Data | Azure Data Factory | Azure DevOps | CI/CD | Data Factory | Data Governance | Databricks | Databricks Workflows | Delta Lake | Delta Live Tables | Delta Live) | Delta Optimizations | Dimensional Modeling | Docker | ELT | ETL | Feature Engineering | Git | GitHub Actions | Kubernetes | Lakehouse | MLflow | Oracle | PII masking | Photon | Postgres | PySpark | SQL Server | Schema evolution | Serverless compute | Snowflake | Spark SQL | Structured Streaming | Terraform | Third Normal Form | Unity Catalog | Z-Order
Education
Bachelor of Engineering | Bachelor of Science | Bachelor of Science in Computer Science | Bachelor of Science in Data Science
Roles
Related jobs
-
Lead Data Engineer USD 125K-160KBigQuery | CI/CD | Cloud Composer | Cloud Storage | Cloud platform401k | Commuter benefits | Dental insurance | Disability coverage | Employee assistance programSenior-level Full TimeColumbia, MD, United States8h ago
-
AI Search | Application Insights | Application diagnostics | Azure AI | Azure AI Search401k | Dental insurance | Educational assistance | Health savings account | Life insuranceMid-level Full TimeNew York, United States R12h ago
-
API | Agile | Data Modeling | Data Quality | Data TransformationCI Polygraph sponsorship within first year | Obtain A Plus within 6 months | Obtain Network Plus within 6 months | Obtain Security Plus within 6 months | TS SCI clearance sponsorshipMid-level Full TimeSpringfield, VA; Falls Church, VA; Falls …13h ago
-
Intelligent Systems Engineer II USD 98K-166KAPIs | AWS Lambda | AWS S3 | Amazon Web Services | Artificial Intelligence401k match | Community service day | Dental insurance | Flexible spending accounts | Life insuranceMid-level Full TimeWest Chester, OH, United States13h ago
-
Intelligent Systems Engineer II USD 98K-166KAPI Gateway | APIs | AWS | Artificial Intelligence | Automated testing401k match | Community service day | Flexible spending accounts | Life insurance | Medical/Dental/Vision insuranceMid-level Full TimeWest Chester, OH, United States13h ago
-
Senior Backend Engineer USD 160K-220KAWS | Asynchronous processing | CI/CD | Cloud infrastructure | Debugging401k access | Career growth | Flexible PTO | Health, dental, vision coverage | Paid mealsSenior-level Full TimeSan Francisco, CA, US15h ago
-
Senior DevOps Engineer (Storage) - remote in the US USD 200K-300KAlerting | ArgoCD | CAPI | CSI | Cluster APICompany events | Conference attendance | Hackathons | Professional development and training | Tech talksExecutive-level Full TimeRemote, USA, United States R15h ago
-
Data Ops Engineer USD 130K-170KCI/CD | Incident Management | Log Management | Monitoring | ObservabilityOnsite work scheduleSenior-level Full TimeErie, PA, United States16h ago
-
AI Architect USD 111K-139KAI Search | Azure | Azure AI | Azure AI Search | Azure DevOps401k match | Employee discounts | Financial wellness planning | Great Place to Work certified | Health, dental and vision coverageSenior-level Full TimeCincinnati, Ohio, USA17h ago
-
AI Architect USD 148K-185KAI Search | Authentication | Azure | Azure AI | Azure AI Search401k match | Employee discounts | Financial wellness planning | Great Place to Work certification | Health, dental and vision coverageSenior-level Full TimeTampa, Florida, USA17h ago
-
Senior Software Engineer (Storage) - remote in the US USD 100K-175KArgoCD | CSI | FluxCD | GitOps | GoSenior-level Full TimeRemote, USA, United States R17h ago
-
Software Engineer III, AI/ML Machine Learning, Core USD 147K-210KAI/ML | AI/ML model development | Artificial Intelligence | C++ | CI/CDSenior-level Full TimeAustin, TX, USA18h ago
-
Senior-level Full TimeTel Aviv-Yafo, Tel Aviv District, IL20h ago
-
Staff Machine Learning Engineer, Gen AI | Voice & Speech USD 150K-220KBigtable | Cloud Computing | Cloud Native | Cloud-native infrastructure | Data integrationRemote US basedSenior-level Full TimeUS Remote R1d ago
-
Sr. Lead Software Engineer - Data Engineer USD 170K-186KAWS | Agentic AI | Agile | Apache Iceberg | Best practicesSenior-level Full TimeJersey City, NJ, United States1d ago
-
.NET | AI Foundry | Agile | Angular | App Service401 K | Employee stock ownership plan | Free hot lunches | Health, dental, vision insurance | Long-term disability insuranceSenior-level Full TimeLakeland, FL, Florida, 333 E Lemon …1d ago
-
Senior Data Analytics Consultant USD 113K-188KData Integrity | Data Security | Data Visualization | Data pipeline | ETL401k retirement plan | Adoption Assistance | Dental insurance | Employee referral program | Flexible spending accountSenior-level Full TimeClient Office: Huntsville, AL, United States1d ago
-
Senior Data Engineer USD 28KAgentic AI | Agile | Apache Iceberg | CI/CD | Confluence401k plan | Commuter benefits | Disability benefits | Life insurance | Paid time offSenior-level Full Time142019-NC-300 South Brevard, Charlotte, United States1d ago
-
API Development | Accessibility | Apache JMeter | Apache Kafka | Apache Spark401k match | Adoption Assistance | Healthcare benefits | Life insurance | Long-term disabilitySenior-level Full TimeIrving, TX, United States R1d ago
-
Senior UX Developer (NIH Data Lakehouse Support) USD 110K-149KAWS | Azure | CI/CD | CSS | DatabricksPublic trust required clearance process | Remote workSenior-level Full TimeUSA MD Home Office (MDHOME), United … R1d ago
-
Senior AI Engineer (Remote) USD 100K-250KAPI Integration | Agent systems | Agentic Reasoning | Automated rollback | Automated testing401k | ESPP | Health care benefits | Paid time off | Success sharing bonusSenior-level Full TimeSTORE SUPPORT CENTER, ATLANTA - 9090, … R1d ago
-
Lead Engineer - AI Engineering USD 89K-148KGPT | Git | Hugging Face | Language Models | Language Processing401k match | Disability benefits | Employee assistance program | Health Coach | Life insuranceSenior-level Full TimeNiskayuna, United States R1d ago
-
AI Solutions Specialist USD 105K-180KAI Governance | Access Control | Artificial Intelligence | Azure AI | Cloud AI401k matching | Career Development Programs | Dental insurance | Employee assistance program | Life insuranceMid-level Full TimeUSA-MI-Ann Arbor-KLA, United States1d ago
-
API Integration | Algorithms | Automation | C++ | CI/CDSenior-level Full TimeChicago, IL, USA, United States1d ago
-
Marketing Analytics Engineer, Data Products & Activation USD 127K-160KAI Governance | Analytics engineering | Attribution Modeling | CRM Integration | Data ContractsMid-level Full TimeUS MA Boston, United States1d ago