ETL/Data Engineer
Indianapolis, IN, United States
USD 135K-165K (estimate) Senior-level Full Time
Tasks
- Build PySpark notebooks and jobs
- Define and enforce engineering coding standards
- Design and build data ingestion pipelines
- Design dimensional models and data vault patterns
- Develop and optimize ELT workflows
- Implement CI/CD for data pipelines
- Implement medallion architecture
- Implement slowly changing dimensions and change data capture
- Instrument pipelines with logging auditing and monitoring
- Lead legacy to cloud data migrations
- Monitor and troubleshoot data pipeline performance
- Perform workload assessment capacity planning and cost modeling
- Respond to production incidents for critical pipelines
- Tune distributed SQL workloads
Perks/Benefits
- N/A
Skills/Tech-stack
ACLs | Active Directory | Azure Active Directory | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2 | Azure Databricks | Azure DevOps | Azure Key Vault | Azure Monitor | Azure Resource | Azure Resource Manager | Azure SQL | Azure Synapse | Azure Synapse Analytics | Bicep | CI/CD | Data Factory | Data Lake Storage Gen2 | Dedicated SQL Pool | Delta Lake | Entra ID | Hierarchical Namespace | KQL | Key Vault | Lifecycle Management | Log Analytics | Managed Identities | Microsoft Fabric | Network Integration | Pandas | Parquet | Private Endpoints | PySpark | Pytest | Python | RBAC | REST API | Resource manager | SQL | SQL Pool | Service principals | Spark SQL | Synapse Analytics | Synapse Dedicated SQL Pool | T-SQL | Teradata | Terraform | Unit Testing | Unity Catalog | Virtual Network | Virtual Network Integration | YAML
Education
N/A
Related jobs
-
AI Solutions Specialist USD 148K-200KAI Search | Agent Frameworks | Artificial Intelligence | Automation | Azure AIHybrid workMid-level Full TimeKansas City, MO, United States6h ago
-
Data Science Engineer (Shreveport, LA) USD 37K-40KData Historian | Data Visualization | Data analytics | Excel | Machine Learning401k match | Dental insurance | Disability insurance | Health insurance | Life insuranceMid-level Full TimeAtlanta, GA, United States R6h ago
-
AWS Bedrock | CI/CD | CUI SPI | Chunking | Data GovernanceGovernment furnished equipment | No sponsorship | No subs requirement | Remote workEntry-level Full TimeUnited States8h ago
-
Staff Software Engineer, Knowledge Catalog, AI USD 207K-301KBig Data | Computer Vision | Context engineering | Conversational Analytics | Data GovernanceBenefits | Bonus | EquitySenior-level Full TimeSunnyvale, CA, USA; Kirkland, WA, USA12h ago
-
Senior Data Engineer USD 156K-227KApache Flume | Apache Spark | Data Analysis | Data Engineering | Data ModelingSenior-level Full TimeMountain View, CA, USA; Zürich, Switzerland12h ago
-
Lead Analyst Data Engineer USD 129K-170KAWS Glue | AWS Lake Formation | AWS Step Functions | Alation | Amazon AthenaSenior-level Full TimeHouston, TX, 77046, US12h ago
-
Principal Research Data Engineer USD 142K-185KAirflow | Analytical processing | ArcGIS | Avro | CI/CDDental | Health care | PTO | Retirement | Sick leaveSenior-level Full TimeSt. Louis, Missouri, US R15h ago
-
Generative AI Engineer - Ford Pro Intelligence USD 118K-232KAPI Integration | Agent Orchestration | Cloud deployment | Cloud platform | Data OutputsMid-level Full TimeBellevue, WA, United States18h ago
-
Machine Learning Engineer (Active Secret Clearance) USD 160K-190KAgile | Asynchronous programming | CI/CD | Data Engineering | Docker401k plan | FSA | Fully remote work | HSA | Hybrid onsite optionMid-level Full TimeRemote; Tacoma, WA R20h ago
-
HR AI/ML Engineer II USD 90K-135KApache Airflow | Databricks | ETL | Git | Machine LearningDental insurance | Employee assistance program | Employee stock ownership plan | Health insurance | Paid parental leaveSenior-level Full TimeHerndon, VA, United States23h ago
-
Analytics platform | Azure Data | Azure Data Factory | Azure Databricks | Data Architecture401k | Dental insurance | Health insurance | Holiday pay | Sick leaveMid-level Full TimeNew York - 300 Madison Avenue, …23h ago
-
Cloud Storage | Compute Orchestration | Computer Vision | Data Lineage | Data PipelinesEnd-to-end responsibility | Fast-paced startup environment | High autonomy | Onsite work | OwnershipMid-level Full TimeSan Mateo, CA; Onsite R23h ago
-
Market Data Platform Engineer USD 168KAccess Control | Automation | Bash | Capacity Planning | Disaster Recovery401k plan | Disability benefits | Life insurance | Paid time off | Parental leaveMid-level Full Time142019-NC-300 South Brevard, Charlotte, United States23h ago
-
Principal Optimization Engineer USD 117K-234KCONOPT | Cloud Computing | Convergence analysis | Discrete Optimization | Fluid modelingHealth care benefits | Hybrid remote option | Paid Holidays | Paid sick days | Paid vacationSenior-level Full TimeCAG10: ALC HQ, 1025 Cobb Place … R23h ago
-
Data/AI Engineer USD 94K-135KACID | Artificial Intelligence | Automated testing | Azure DevOps | Bronze Silver Gold401k savings plan | Flexible Spending Accounts FSA | Health Savings Account (HSA) | Healthy Lifestyle Programs | Long-term disabilitySenior-level Full TimeUS-Nationwide-FIELD, United States23h ago
-
Data AI Engineer, Dir, P4 USD 150K-190KAWS | Apache Spark | Azure | Data Architecture | Data GovernanceHybrid workSenior-level Full Time1 New York Plaza, United States23h ago
-
Analytics Engineer USD 118K-177KAWS | Airflow | DBT | Dagster | Data Architecture401k match | Dental insurance | Life insurance | Medical insurance | Tuition reimbursementSenior-level Full TimeWaltham Office, United States23h ago
-
Senior Big Data Engineer USD 130K-196KAWS | Access Control | Airflow | Alerting | Apache Spark401k matching | Employee assistance program | Flexible paid time off | Medical, dental, vision, life, and disability insurance | Paid HolidaysSenior-level Full TimeSan Francisco, United States23h ago
-
Pathology Computational Scientist USD 93K-136KBioinformatics | Containerization | Data Analysis | Data Visualization | Database systemsCareer advancement opportunities | Comprehensive benefits | Onsite work environmentMid-level Full Time60 Fenwood Road Boston (Hale Building), …23h ago
-
APIs | Agile | Azure | Azure Data | Azure Data FactoryPeriodic travel | Remote work permittedSenior-level Full Time6314 Remote/Teleworker US, United States R23h ago
-
Senior-level Full TimePlano, TX, United States23h ago
-
C++ | Embedded Linux | Hardware-in-the-loop | Hardware-in-the-loop Testing | Loop Testing401k match | Education reimbursement | Health Savings Account contribution | Paid time offMid-level Full TimeCambridge - B3 Crossing, United States23h ago
-
Principal Storage & Disaster Recovery Engineer USD 109K-223KAWS EBS | AWS EFS | AWS FSX | AWS S3 | Azure BlobSenior-level Full TimeBusiness Office - Benson Tower, United …23h ago
-
Robotics Forward Deployed Engineer USD 135K-217KAutomation | Cause analysis | Control Systems | Dashboards | Data VisualizationCross-functional collaboration | On site work experience | Travel to customer sitesSenior-level Full TimeWatertown, MA1d ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudAsynchronous culture | Bonus | Equity | Friendly and laid-back atmosphereMid-level Full TimeSeattle, WA, USA1d ago