Lead Software and Data Engineer
Chicago, Illinois, United States
USD 155K-200K Senior-level Full Time
Tasks
- Architect and maintain scalable data lake on ADLS Gen2
- Collaborate with stakeholders to ensure accurate accessible structured data
- Design ETL and ELT pipelines and schema governance
- Design build and maintain production data pipelines
- Develop Kubernetes based microservices for data and API workloads
- Establish code review CI CD testing and documentation standards
- Implement security and data governance protocols
- Migrate data processes from Azure Synapse to Dagster
- Own Dagster orchestration assets sensors schedules and deployments
- Propose and prototype AI enabled solutions with rapid turnaround
- Support PySpark workloads with DataFrames and Spark SQL
Perks/Benefits
- 401k
- Disability insurance
- Life insurance
- Medical/Dental/Vision
- Paid time off
- Parental leave
- Profit sharing
Skills/Tech-stack
ADLS Gen2 | Apache Spark | Azure | Azure Data | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2 | Azure Synapse | CI/CD | Dagster | Dagster Cloud | Data Lake | Data Lake Storage Gen2 | Data Modeling | Dataframes | ELT | ETL | GitHub Copilot | Kubernetes | Machine Learning | PySpark | Python | SQL | Spark SQL | Test automation
Education
Bachelor of Engineering | Bachelor of Engineering in Software Engineering | Bachelor of Science | Bachelor of Science in Computer Science
Roles
Data Engineer | Engineer | Lead | Lead Data | Lead Data Engineer | Lead Software Engineer | Software Engineer
Related jobs
-
Senior-level Full TimeUS - San Francisco6h ago
-
AWS | Azure | CRM | Data pipeline | DatabricksSenior-level Full TimeNew York, NY, US R11h ago
-
Data Engineer, Data Platform - Distribution USD 180K-210KAccess Control | Agentic Systems | Auditing | BI Modernization | Business IntelligenceSenior-level Full TimeBoston, MA11h ago
-
FSOC Embedded Controls Engineer USD 175K-215KAdaptive optics | Closed Loop | Closed loop control | Control Systems | DSP401k matching | Dental insurance | Equity options | Health insurance | Hybrid remote and in office scheduleMid-level Full TimeLivermore, CA12h ago
-
Software Development Engineer - Robotics USD 100K-170KC++ | CUDA | CUDNN | GPU Acceleration | IMUCareer growth opportunities | Comprehensive benefits | MentorshipMid-level Full TimeBoston, Massachusetts12h ago
-
Data Engineer, Expert USD 133K-238KApache Spark | Big Data | CI/CD | Cloud Data | Cloud Data WarehouseSenior-level Full TimeOakland, CA, US, 9461212h ago
-
Data Engineer II USD 63K-114KETL | SQL | Snowflake401k | Flexible work schedule | Health insurance | Paid time off | Tuition reimbursementMid-level Full TimeRemote-MN, United States R12h ago
-
Senior Software Engineer, Data Platform USD 128K-168KAWS | Amazon EC2 | Amazon RDS | Amazon S3 | ClickHouseGenerous vacation policies | Health insurance coverage | Paid leave | Relocation assistanceSenior-level Full TimeBerkeley, CA12h ago
-
Staff Software Engineer, Data Platform USD 147K-194KAmazon Web Services | ClickHouse | DBT | Dagster | DatabricksMedical/Dental/Vision insurance | Paid leave | Paid parental leave | Relocation assistance | Vacation timeSenior-level Full TimeBerkeley, CA12h ago
-
Autonomous Vehicles | C++ | Data Pipelines | Deep learning | JAXCompany benefits program | Discretionary annual bonus | Equity incentive planSenior-level Full TimeMountain View, CA, USA; San Francisco, …12h ago
-
Senior Machine Learning Engineer USD 139K-227KAudio Processing | CUDA | ChromaDB | Computer Vision | Distributed Computing401k match | Continuing education support | Function health subscription | Health & wellness stipend | Health, dental, vision benefitsSenior-level Full TimeAustin, TX12h ago
-
Software Engineer Intern, Perception Data USD 114K-114K2D Bounding Boxes | 3D Bounding Boxes | AWS S3 | Airflow | AngularHousing stipend | Medical insurance | Relocation assistanceEntry-level InternshipFoster City, CA13h ago
-
Mid-level Full TimeNew York13h ago
-
Senior Software Engineer, Data Governance USD 130K-250KAWS | Access Control | Airflow | Amazon Bedrock | Amazon SageMakerBonus | Equity | Health insurance | Open to office work in San FranciscoSenior-level Full TimeSan Francisco, CA13h ago
-
Sr. AI / Embedded ML Engineer USD 120K-200KARM Cortex | ARM Cortex-M | Accelerometer | Automl | BarometerContinuous learning and development | Fair and honest workplace | Financial solutions for the future | Health and wellness care options | Optional Legal Services US OnlySenior-level Full TimeSaratoga, CA13h ago
-
Principal Data Engineer USD 160K-200KAI Agents | AWS Glue | AWS Lambda | Amazon ECS | Amazon MSK401k | Dental insurance | Discounts | Fully remote | Medical insuranceExecutive-level Full TimeNew York, NEW YORK, United States R13h ago
-
Senior Database Engineer USD 150K-180KApache Spark | Azure Data | Azure Data Factory | Azure Data Lake | Azure Databricks401k company contribution | Career progression | Dental insurance | Flexible vacation policy | Health insuranceSenior-level Full TimeRemote (United States) R14h ago
-
Software Engineer, Research Data Platform USD 320K-405KAPIs | Application development | BigQuery | Data Cataloging | Data LineageFlexible working hours | Generous vacation | Hybrid work policy | Optional equity donation matching | Parental leaveMid-level Full TimeSan Francisco, CA | New York …14h ago
-
Staff Engineer – Experimentation Team USD 182K-295KAWS | Bayesian | Bayesian optimization | BigQuery | CUPEDDental insurance | Health insurance | Mental health benefits | Restricted stock units | Vision insuranceSenior-level Full TimeRemote - US R14h ago
-
AWS | Algorithms | Azure | C# | ContainersMid-level Full TimeBellevue14h ago
-
Computational Geometry Senior Software Engineer USD 120K-165K3D CAD | 3D Modeling | 3D engineering analysis | 3D scanning | ACISHybrid work | Ownership planSenior-level Full TimeUnited States14h ago
-
Senior Software Engineer, Data Engineering USD 130K-150KApache Kafka | Apache Spark | At least once | Automated testing | AvroSenior-level Full TimeAndover, Massachusetts, United States15h ago
-
Senior Software Engineer, Data Engineering USD 121K-161KApache Kafka | At least once | At least once semantics | Avro | BackpressureSenior-level Full TimeCarlsbad, California, United States15h ago
-
Senior Software Engineer, Data Engineering USD 121K-161KApache Spark | At least once | At least once semantics | Automated testing | AvroSenior-level Full TimeManhattan, New York, United States15h ago
-
GenAI Product Engineer USD 70K-136KAPI Development | Cloud Computing | Cloud platform | DevOps | Generative AICertification training | Fully remote within contiguous US | Hands-on experienceMid-level Full TimeArlington, VA R15h ago