ETL/Data Engineer
Indianapolis, IN, United States
USD 135K-165K (estimate) Senior-level Full Time
Tasks
- Build PySpark notebooks and jobs
- Define and enforce engineering coding standards
- Design and build data ingestion pipelines
- Design dimensional models and data vault patterns
- Develop and optimize ELT workflows
- Implement CI/CD for data pipelines
- Implement medallion architecture
- Implement slowly changing dimensions and change data capture
- Instrument pipelines with logging auditing and monitoring
- Lead legacy to cloud data migrations
- Monitor and troubleshoot data pipeline performance
- Perform workload assessment capacity planning and cost modeling
- Respond to production incidents for critical pipelines
- Tune distributed SQL workloads
Perks/Benefits
- N/A
Skills/Tech-stack
ACLs | Active Directory | Azure Active Directory | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2 | Azure Databricks | Azure DevOps | Azure Key Vault | Azure Monitor | Azure Resource | Azure Resource Manager | Azure SQL | Azure Synapse | Azure Synapse Analytics | Bicep | CI/CD | Data Factory | Data Lake Storage Gen2 | Dedicated SQL Pool | Delta Lake | Entra ID | Hierarchical Namespace | KQL | Key Vault | Lifecycle Management | Log Analytics | Managed Identities | Microsoft Fabric | Network Integration | Pandas | Parquet | Private Endpoints | PySpark | Pytest | Python | RBAC | REST API | Resource manager | SQL | SQL Pool | Service principals | Spark SQL | Synapse Analytics | Synapse Dedicated SQL Pool | T-SQL | Teradata | Terraform | Unit Testing | Unity Catalog | Virtual Network | Virtual Network Integration | YAML
Education
N/A
Related jobs
-
Bash | Data Processing | Docker | GCP | Infrastructure as CodeAsynchronous culture | Career growth opportunities | Remote or distributed workMid-level Full TimeBellevue, WA, USA2h ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudBonus | Equity | Friendly laid-back atmosphere | Remote friendly asynchronous cultureMid-level Full TimeDenver, CO, USA2h ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudAsynchronous culture | Flexible environment | Inclusive workplace | Remote-friendlyMid-level Full TimeAnaheim, CA, USA2h ago
-
Bash | Docker | GCP | Linux | PythonAsynchronous work culture | Friendly laid-back atmosphere | Hands-off managementMid-level Full TimeProvo, UT, USA2h ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudAsynchronous culture | Flexible work environment | Remote/distributed workMid-level Full TimeFort Worth, TX, USA2h ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudAsynchronous culture | Bonus | Equity | Friendly and laid-back atmosphereMid-level Full TimeHouston, TX, USA2h ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudAsynchronous culture | Flexible team environment | Remote-friendlyMid-level Full TimeCollege Park, MD, USA2h ago
-
Bash | Cloud infrastructure | Cloud platform | Data Processing | DockerAsynchronous culture | Friendly atmosphere | Remote or distributed workMid-level Full TimeSaint Paul, MN, USA2h ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudAsynchronous culture | Collaborative team | Remote friendly distributed workMid-level Full TimeDurham, NC, USA2h ago
-
Bash | Data Ingestion | Data Ingestion Pipelines | Data Processing | DockerAsynchronous culture | Career impact | Laid-back atmosphere | Remote-friendly | Supportive teamMid-level Full TimeCary, NC, USA2h ago
-
Bash | Cloud infrastructure | Cloud platform | Data Ingestion | Data ProcessingAsynchronous work culture | Flexible management style | Friendly laid-back atmosphereMid-level Full TimeCleveland, OH, USA2h ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudAsynchronous culture | Bonus opportunity | Equity opportunity | Friendly work environmentMid-level Full TimeSt. Louis, MO, USA2h ago
-
Bash | Cloud platform | Data Ingestion | Data Processing | DockerAsynchronous culture | Friendly laid-back atmosphere | Portfolio and LinkedIn application support | Remote distributed settingMid-level Full TimeCambridge, MA, USA2h ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudAsynchronous culture | Career growth opportunities | Flexible work environment | Remote-friendlyMid-level Full TimeArlington, VA, USA2h ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudAsynchronous culture | Flexible remote workMid-level Full TimeReston, VA, USA2h ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudFlexible asynchronous culture | Friendly and laid-back atmosphere | Remote friendly distributed work environmentMid-level Full TimeSpokane, WA, USA2h ago
-
Bash | Data Processing | Docker | GCP | Infrastructure as CodeAsynchronous culture | Entrepreneurial team environment | Opportunities for impact | Remote-friendly work cultureMid-level Full TimeReno, NV, USA2h ago
-
Bash | Data Processing | Docker | GCP | Infrastructure as CodeAsynchronous culture | Bonus | Competitive salary | Equity | Remote workMid-level Full TimeSanta Cruz, CA, USA2h ago
-
Bash | Data Processing | Docker | GCP | Large Scale DataAsynchronous culture | Flexible remote work | Friendly laid-back atmosphere | Impactful product missionMid-level Full TimeNew Orleans, LA, USA2h ago
-
Bash | Data Processing | Docker | GCP | Infrastructure as CodeMid-level Full TimeWest Palm Beach, FL, USA2h ago
-
Bash | Data Ingestion | Data Processing | Docker | GCPAsynchronous culture | Friendly work environment | Remote distributed work environmentMid-level Full TimeOklahoma City, OK, USA2h ago
-
Bash | Cloud platform | Data Ingestion | Data Processing | DockerMid-level Full TimeIthaca, NY, USA2h ago
-
Bash | Data Processing | Docker | GCP | Infrastructure as CodeAsynchronous work culture | Bonus | Competitive salary | Equity | Friendly work environmentMid-level Full TimeBurlington, VT, USA2h ago
-
Bash | Data Processing | Docker | GCP | Large Scale DataAsynchronous culture | Friendly work environment | Hands-off management | Remote/distributed teamMid-level Full TimeCharleston, SC, USA2h ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudAsynchronous culture | Flexible work environment | Impactful mission product | Remote-friendly, distributed teamMid-level Full TimeMemphis, TN, USA2h ago