Senior Data Engineer
Philadelphia, Pennsylvania, United States
USD 130K-165K (estimate) Senior-level Full Time
Tasks
- Build batch and streaming ETL and ELT workflows
- Deliver pipelines using software engineering standards documentation testing and CI/CD
- Design data pipelines for real time streaming
- Develop data services APIs and microservices
- Ensure data quality validation and privacy
- Implement streaming ingestion with Azure Event Hubs
- Optimize Databricks pipelines with PySpark and Spark SQL
- Process streams with Azure Stream Analytics
- Troubleshoot and resolve pipeline issues across systems
- Tune Spark for performance reliability and cost
Perks/Benefits
Skills/Tech-stack
Apache Spark | Azure | Azure Event | Azure Event Hubs | Azure Stream | Azure Stream Analytics | CI/CD | Databricks | Delta Lake | ELT | ETL | Event Hubs | Microservices | PySpark | REST APIs | Spark SQL | Stream Analytics
Education
Roles
Regions
Countries
States
Related jobs
-
Senior-level Full TimeNew York, NY8h ago
-
LA Kings - Sr. Data Engineer USD 111K-135KAlerting | Apache Spark | Batch Processing | Cost Optimization | Data Governance401k match | Health savings account | Life insurance | Paid Holidays | Parental leaveSenior-level Full TimeEl Segundo, CA9h ago
-
Data Analytics Integrator USD 92K-158KData Engineering | Data Modeling | Data Pipelines | Data Quality | Data Reconciliation401k | Dental insurance | Discretionary PTO | Health insurance | Professional development opportunitiesMid-level Full TimeAtlanta, GA10h ago
-
Software Engineer, Machine Learning USD 150K-215KAPI Design | AWS | AWS CloudWatch | Artificial Intelligence | AzurePartial telecommutingMid-level Full TimeBoston, MA12h ago
-
Senior Databricks Platform Engineer USD 180K-247KActive Directory | Apache Hive | Apache Spark | Auditing | AutoscalingPublic Trust DHS EODSenior-level Full TimeArlington, VA, United States13h ago
-
Mid-level Full TimeSioux Falls, SD, United States14h ago
-
Senior Data Engineer USD 135K-180KAKS | Automated testing | Azure Cosmos | Azure Cosmos DB | Azure DataSenior-level Full TimeAtlanta, GA | Kansas City, MO …15h ago
-
Apache Spark | Data Lineage | Data Modeling | Data Quality | Data Quality ManagementRemote work | W2 employmentMid-level Full TimeChicago, IL. R16h ago
-
Data Modeling, Fraud Analytics Consulting (Remote) USD 104K-174KAWS | Data Modeling | Data Science | Fraud Detection | Machine LearningDental insurance | Flexible time off | Flexible work environment | Hybrid work | Medical insuranceMid-level Full TimeUnited States, UNITED STATES, United States R17h ago
-
AI Engineer USD 141K-236KAzure OpenAI | Azure Security | Azure Security Center | Azure Synapse | DatabricksDisability insurance | Health insurance | Holiday pay | Learning and development | Life insuranceSenior-level Full TimeUSA-DC-Washington17h ago
-
Senior Machine Learning Engineer (Multiple Positions) USD 194K-355KApache Spark | Big Data | Data Preparation | Deep learning | Experiment designSenior-level Full TimeSeattle, Washington, United States18h ago
-
Senior Embedded Software Engineer - Flight Software USD 100K-200KBare Metal | Bash | C# | C++ | CANSenior-level Full TimeSaratoga, CA1d ago
-
Senior Machine Learning Engineer (Computer Vision & AI) USD 149K-198KAgile Development | Anomaly Detection | Azure | CUDA | Cloud ComputingTraining opportunitiesSenior-level Full TimeAUBURN HILLS HQ R&D, MI, US1d ago
-
CI/CD | Fastlane | Git | Multithreading | SOLIDAsynchronous culture | Fast-paced environment | Learning Support | Remote workSenior-level Full TimeKansas City, MO, USA1d ago
-
Strategy & Execution AI Specialist USD 92K-127KAPI Integration | BI tools | Data Engineering | Data Pipelines | DatabricksMid-level Full TimeUSCA1d ago
-
Data Engineer, Marketing Data Operations USD 90K-125KAPI | AWS | Amazon EC2 | Amazon Redshift | Amazon S3Mid-level Full TimeOrlando1d ago
-
Engineer in Residence - Generative AI USD 140K-220KAI Security | AWS | CSS | Cloud Architecture | Cloud platformSenior-level Contract Full TimeMountain View, CA1d ago
-
Senior Staff AI Data Engineer - Hybrid USD 135K-202KAWS | Alerting | Azure | Batch Processing | ChunkingCommunity of Practice | Hybrid work schedule | Mentorship opportunitiesSenior-level Full TimeHartford CT- Home Office, United States R1d ago
-
Software Engineer (AI/ML) USD 110K-177KAWS | Agile | Azure | Docker | GCP401k match | Dental insurance | Employee assistance program | Health insurance | Paid parental leaveMid-level Full TimeBoston, United States R1d ago
-
Prin AI & Analytics Data Architect USD 160K-235KAPIs | AWS | BI | Cloud Computing | DSPM401k matching | Education reimbursement | Flexible work options | Health and well-being programs | Life and disability insuranceSenior-level Full TimeUnited States - Shakopee, MN (GHQ), …1d ago
-
System Dev Engineer I, Analytics ADC USD 99K-160KAWS | CI/CD | Golang | Infrastructure as Code | Java401k matching | Employee assistance program | Flexible spending accounts | Health insurance | Paid time offMid-level Full TimeDenver, Colorado, USA1d ago
-
DAX | Data Warehousing | Data integration | Dimensional Modeling | ETL401k company match | Disability benefits | Employee assistance program | Employee volunteer program | Flexible spending accountMid-level Full TimeBirmingham, AL - Regions Center, United …1d ago
-
Data Engineer AI USD 165K-216KAWS Glue | AWS Lambda | AWS S3 | AWS Step Functions | AirflowWork-life balanceSenior-level Full TimeTelecommuter TN, United States R1d ago
-
Senior AI/ML Software Engineer USD 121K-195K.NET | APIs | AWS | Agile | Azure401k retirement plan | Employee assistance program | Flexible spending account | Flexible work environment | Health savings accountSenior-level Full TimeRemote, United States, United States R1d ago
-
Principal Integration Engineer - Trade Data Warehouse USD 143K-169KApache Spark | Automation | Data Modeling | Data Pipelining | Docker401k matching | Adoption Assistance | Dental insurance | Disability insurance | HealthcareSenior-level Full TimeCharlotte, NC, United States1d ago