Senior Data Engineer
Philadelphia, Pennsylvania, United States
USD 130K-165K (estimate) Senior-level Full Time
Tasks
- Build batch and streaming ETL and ELT workflows
- Deliver pipelines using software engineering standards documentation testing and CI/CD
- Design data pipelines for real time streaming
- Develop data services APIs and microservices
- Ensure data quality validation and privacy
- Implement streaming ingestion with Azure Event Hubs
- Optimize Databricks pipelines with PySpark and Spark SQL
- Process streams with Azure Stream Analytics
- Troubleshoot and resolve pipeline issues across systems
- Tune Spark for performance reliability and cost
Perks/Benefits
Skills/Tech-stack
Apache Spark | Azure | Azure Event | Azure Event Hubs | Azure Stream | Azure Stream Analytics | CI/CD | Databricks | Delta Lake | ELT | ETL | Event Hubs | Microservices | PySpark | REST APIs | Spark SQL | Stream Analytics
Education
Roles
Regions
Countries
States
Related jobs
-
API | AWS Glue | Amazon Redshift | Amazon Web Services | Apache AirflowFully remote | Mentorship opportunities | On-call supportSenior-level Full TimeOrlando, FL, United States R13h ago
-
Senior Data Engineer USD 140K-160KAirflow | Apache Spark | Auto Loader | CI/CD | Change Data Capture401k | Paid leave | Tuition reimbursementSenior-level Full TimeNorth Hollywood, CA, United States13h ago
-
Mid-level Full TimeMiami Lakes, FL, United States15h ago
-
AWS | Autogen | Automated testing | Azure | CI/CDOnsite workSenior-level ContractDallas, United States19h ago
-
Databricks Engineer USD 97K-162KAPI Integration | Agile | Data Governance | Data Modeling | Data QualityMid-level Full TimeArlington/Rosslyn, Virginia, United States; Baltimore, Maryland, …20h ago
-
Data Scientist, Innovation Hub USD 108K-108KAudit Logging | Azure | CI/CD | Federated identity | Integration TestingCareer growth | Healthcare | On-site childcare | PTO | Paid HolidaysMid-level Full TimeTexas-Dallas-5323 Harry Hines Blvd20h ago
-
Databricks Engineer USD 107K-179KAPI Integration | Agile | Cloud Computing | Data Governance | Data ModelingTravel 10 to 15 percentMid-level Full TimeArlington/Rosslyn, Virginia, United States; Baltimore, Maryland, …20h ago
-
Senior Data Engineer USD 133K-158KAKS | Azure Data | Azure Data Factory | Azure Data Lake | Azure DatabricksSenior-level Full TimeCincinnati, Ohio, United States20h ago
-
AI Program Lead USD 159K-265KAI Governance | Agent-Based Artificial Intelligence | Agent-based | Artificial Intelligence | Business IntelligenceMentorship | Professional development | Travel up to 25 percentSenior-level Full TimeArlington/Rosslyn, Virginia, United States20h ago
-
Associate BI Analytics Engineer USD 98K-163KDAX | Data Modeling | Data Warehousing | Databricks | ELTMid-level Full TimeUSA-VA-Herndon20h ago
-
Senior Software Engineer - Data & Analytics Federation USD 160K-240KApache Arrow | Asynchronous frameworks | C++ | Distributed Systems | Linux401k match | Dental insurance | Life insurance | Medical insurance | Paid HolidaysSenior-level Full TimePrinceton20h ago
-
Data Engineer, Employee Technology Support USD 110K-130KCause analysis | Data Governance | Data Modeling | Data Monitoring | Data Quality401k match | Dental insurance | Life insurance | Long-term disability | Medical insuranceMid-level Full TimeNew York20h ago
-
Security Engineer III - AI/ML USD 170K-215KAPI Development | AWS | Anomaly Detection | Azure | BERTSenior-level Full TimePlano, TX, United States21h ago
-
Principal Data Engineer USD 160K-170KAWS | Amazon Redshift | Apache Airflow | BigQuery | Cloud platform401k match | Dental insurance | Health insurance | Paid time off | Remote workSenior-level Full TimeRemote (United States) R1d ago
-
AI/ML Engineer - School USD 101K-163KAI Safety | AI safety evaluation | AWS Bedrock | AWS ECS | AWS LambdaMid-level Full TimeVirtual US IL, United States R1d ago
-
AI Software Engineer - Fixed Income Technology USD 110K-150KAPI Integration | AWS | Azure | CI/CD | Data PipelinesHealth insurance | Hybrid work | Life insurance | Paid time off | Retirement planSenior-level Full TimeIL01 - Chicago, United States1d ago
-
Senior Cloud Database Engineer (Hybrid - Seattle, WA) USD 142K-220KASM | AWR | AWS DMS | AWS RDS | Amazon AuroraDisability insurance | EAP Resources | Hybrid work location | Life insurance | Medical/Vision/DentalSenior-level Full TimeSeattle WA, United States R1d ago
-
Senior Machine Learning Engineer USD 159K-295KAUC-ROC | AWS | Causal Inference | Clean room | Data clean roomSenior-level Full TimeCA San Francisco 153 Kearny Street, …1d ago
-
Principal AI Platform Engineer USD 117K-206KAI | AWS | Agent workflows | Ansible | Argo CDHealth insurance | Hybrid work | Paid time off | Remote work | Retirement savings planSenior-level Full TimeUSA - MN - Minneapolis, United …1d ago
-
Apache Iceberg | Atlan | BigQuery | Cloud Composer | Cloud RunSenior-level Full TimePA, Working at Home - Pennsylvania, …1d ago
-
AWS | Ab Initio | Artificial Intelligence | Data Governance | Data Quality401k matching | Health insurance | Mental health counseling | Paid Holidays | Paid time offMid-level Full TimeHartford - Tower, United States1d ago
-
Code review | Data Governance | Data Ingestion | Data Mastering | Data SharingSenior-level Full TimePrimary location: Dublin, Franklin, OH R1d ago
-
Senior Data Platform Engineer USD 109K-156KAPI Integration | Azure Data | Azure Data Factory | Azure SQL | Azure SynapseHybrid scheduleSenior-level Full TimeOrlando Office, United States R1d ago
-
Database & Data Platform Administrator USD 105K-125KAutomation | Backup & Recovery | Backup and Recovery | CDC | CI/CDDeveloper enablement | Hybrid work option | MentorshipSenior-level Full TimeDenver, CO, USA1d ago
-
Senior Engineer, Platform Analytics USD 144K-190KApache Airflow | CDC | CI/CD | Change Data Capture | ClickHouseProfessional development funds | Remote flexibility | Unlimited PTO | Wellness reimbursementSenior-level Full TimeRemote - USA, United States R1d ago