Lead Software and Data Engineer
Chicago, Illinois, United States
USD 155K-200K Senior-level Full Time
Tasks
- Architect and maintain scalable data lake on ADLS Gen2
- Collaborate with stakeholders to ensure accurate accessible structured data
- Design ETL and ELT pipelines and schema governance
- Design build and maintain production data pipelines
- Develop Kubernetes based microservices for data and API workloads
- Establish code review CI CD testing and documentation standards
- Implement security and data governance protocols
- Migrate data processes from Azure Synapse to Dagster
- Own Dagster orchestration assets sensors schedules and deployments
- Propose and prototype AI enabled solutions with rapid turnaround
- Support PySpark workloads with DataFrames and Spark SQL
Perks/Benefits
- 401k
- Disability insurance
- Life insurance
- Medical/Dental/Vision
- Paid time off
- Parental leave
- Profit sharing
Skills/Tech-stack
ADLS Gen2 | Apache Spark | Azure | Azure Data | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2 | Azure Synapse | CI/CD | Dagster | Dagster Cloud | Data Lake | Data Lake Storage Gen2 | Data Modeling | Dataframes | ELT | ETL | GitHub Copilot | Kubernetes | Machine Learning | PySpark | Python | SQL | Spark SQL | Test automation
Education
Bachelor of Engineering | Bachelor of Engineering in Software Engineering | Bachelor of Science | Bachelor of Science in Computer Science
Roles
Data Engineer | Engineer | Lead | Lead Data | Lead Data Engineer | Lead Software Engineer | Software Engineer
Related jobs
-
Senior-level Full TimeUS - San Francisco5h ago
-
Data Engineer, Data Platform - Distribution USD 180K-210KAccess Control | Agentic Systems | Auditing | BI Modernization | Business IntelligenceSenior-level Full TimeBoston, MA10h ago
-
Data Engineer, Expert USD 133K-238KApache Spark | Big Data | CI/CD | Cloud Data | Cloud Data WarehouseSenior-level Full TimeOakland, CA, US, 9461210h ago
-
Senior Software Engineer, Data Platform USD 128K-168KAWS | Amazon EC2 | Amazon RDS | Amazon S3 | ClickHouseGenerous vacation policies | Health insurance coverage | Paid leave | Relocation assistanceSenior-level Full TimeBerkeley, CA10h ago
-
Staff Software Engineer, Data Platform USD 147K-194KAmazon Web Services | ClickHouse | DBT | Dagster | DatabricksMedical/Dental/Vision insurance | Paid leave | Paid parental leave | Relocation assistance | Vacation timeSenior-level Full TimeBerkeley, CA10h ago
-
Autonomous Vehicles | C++ | Data Pipelines | Deep learning | JAXCompany benefits program | Discretionary annual bonus | Equity incentive planSenior-level Full TimeMountain View, CA, USA; San Francisco, …11h ago
-
Senior Machine Learning Engineer USD 139K-227KAudio Processing | CUDA | ChromaDB | Computer Vision | Distributed Computing401k match | Continuing education support | Function health subscription | Health & wellness stipend | Health, dental, vision benefitsSenior-level Full TimeAustin, TX11h ago
-
Mid-level Full TimeNew York12h ago
-
Senior Software Engineer, Data Governance USD 130K-250KAWS | Access Control | Airflow | Amazon Bedrock | Amazon SageMakerBonus | Equity | Health insurance | Open to office work in San FranciscoSenior-level Full TimeSan Francisco, CA12h ago
-
Sr. AI / Embedded ML Engineer USD 120K-200KARM Cortex | ARM Cortex-M | Accelerometer | Automl | BarometerContinuous learning and development | Fair and honest workplace | Financial solutions for the future | Health and wellness care options | Optional Legal Services US OnlySenior-level Full TimeSaratoga, CA12h ago
-
Principal Data Engineer USD 160K-200KAI Agents | AWS Glue | AWS Lambda | Amazon ECS | Amazon MSK401k | Dental insurance | Discounts | Fully remote | Medical insuranceExecutive-level Full TimeNew York, NEW YORK, United States R12h ago
-
Software Engineer, Research Data Platform USD 320K-405KAPIs | Application development | BigQuery | Data Cataloging | Data LineageFlexible working hours | Generous vacation | Hybrid work policy | Optional equity donation matching | Parental leaveMid-level Full TimeSan Francisco, CA | New York …13h ago
-
Staff Engineer – Experimentation Team USD 182K-295KAWS | Bayesian | Bayesian optimization | BigQuery | CUPEDDental insurance | Health insurance | Mental health benefits | Restricted stock units | Vision insuranceSenior-level Full TimeRemote - US R13h ago
-
AWS | Algorithms | Azure | C# | ContainersMid-level Full TimeBellevue13h ago
-
Computational Geometry Senior Software Engineer USD 120K-165K3D CAD | 3D Modeling | 3D engineering analysis | 3D scanning | ACISHybrid work | Ownership planSenior-level Full TimeUnited States13h ago
-
Senior Software Engineer, Data Engineering USD 130K-150KApache Kafka | Apache Spark | At least once | Automated testing | AvroSenior-level Full TimeAndover, Massachusetts, United States13h ago
-
Senior Software Engineer, Data Engineering USD 121K-161KApache Kafka | At least once | At least once semantics | Avro | BackpressureSenior-level Full TimeCarlsbad, California, United States13h ago
-
Senior Software Engineer, Data Engineering USD 121K-161KApache Spark | At least once | At least once semantics | Automated testing | AvroSenior-level Full TimeManhattan, New York, United States13h ago
-
GenAI Product Engineer USD 70K-136KAPI Development | Cloud Computing | Cloud platform | DevOps | Generative AICertification training | Fully remote within contiguous US | Hands-on experienceMid-level Full TimeArlington, VA R14h ago
-
Senior Staff Machine Learning Engineer - Agentic Systems USD 281K-401KAgent Orchestration | Data Pipelines | Distributed Systems | Evaluation systems | Experimentation401k retirement plan | Health insurance | Meal allowance | Paid Holidays | Paid parental leaveSenior-level Full TimeNew York, NY14h ago
-
Staff Data Engineer USD 170K-200KABAC | Aerospike | Amazon Redshift | Amazon S3 | Amazon SNSEmployee discounts | Employee equity | Medical, dental, and vision coverage | Pet insurance | Unlimited PTOSenior-level Full TimeRemote - United States R14h ago
-
A/B | A/B Testing | Airflow | B testing | BigQuerySenior-level Full TimeSan Francisco Bay Area14h ago
-
Databricks Lead (Data Migration Lead) USD 180K-248KAWS | Agile | Batch Processing | CI/CD | Data GovernanceOnsite workSenior-level Full TimeDetroit, MI, United States14h ago
-
Tech Lead, Embedded Systems Engineering USD 191K-253KAgile | Architecture | C++ | CI/CD | Code reviewCaregiver and wellness leave | Commuter benefits | Family planning and parenting support | Generous time off | Healthcare benefitsSenior-level Full TimeSeattle, Washington, United States14h ago
-
Staff Machine Learning Engineer USD 220K-280KAWS | Amazon SageMaker | Apache Flink | BigQuery | C++401k match | Annual performance reviews | Company equipment | Dental insurance | Disability benefitsSenior-level Full TimeAtlanta, GA preferred, Remote R14h ago