Senior Data Engineer
Philadelphia, Pennsylvania, United States
USD 130K-165K (estimate) Senior-level Full Time
Tasks
- Build batch and streaming ETL and ELT workflows
- Deliver pipelines using software engineering standards documentation testing and CI/CD
- Design data pipelines for real time streaming
- Develop data services APIs and microservices
- Ensure data quality validation and privacy
- Implement streaming ingestion with Azure Event Hubs
- Optimize Databricks pipelines with PySpark and Spark SQL
- Process streams with Azure Stream Analytics
- Troubleshoot and resolve pipeline issues across systems
- Tune Spark for performance reliability and cost
Perks/Benefits
Skills/Tech-stack
Apache Spark | Azure | Azure Event | Azure Event Hubs | Azure Stream | Azure Stream Analytics | CI/CD | Databricks | Delta Lake | ELT | ETL | Event Hubs | Microservices | PySpark | REST APIs | Spark SQL | Stream Analytics
Education
Roles
Regions
Countries
States
Related jobs
-
Sr Technical Solutions Engineering USD 130K-178KAWS | Automated Patch Deployment | Azure | Bash | CloudFormation24x7 on-call support | Secure facility accessSenior-level Full TimeMcLean, Virginia7h ago
-
AWS | Apache Spark | Big Data | Cloud platform | Data EngineeringSenior-level Full TimeMcLean, Virginia; Washington, D.C.9h ago
-
AWS | Apache Spark | Azure | Cloud platform | Data EngineeringConference and meetup promotion | Equity eligibility | Performance bonus eligibility | Travel opportunitiesSenior-level Full TimeMaryland; Virginia; Washington, D.C.9h ago
-
Staff Software Engineer, AI Native Web Platform USD 198K-272KCI/CD | CMS | Design Systems | JavaScript | LLMSenior-level Full TimeMountain View, California9h ago
-
Senior AI Engineer, Real-World Data USD 125K-175KAI orchestration | AWS | AWS Fargate | AWS Lambda | Agile deliverySenior-level Full TimeUS Remote R9h ago
-
Senior-level Full TimeNew York, NY10h ago
-
LA Kings - Sr. Data Engineer USD 111K-135KAlerting | Apache Spark | Batch Processing | Cost Optimization | Data Governance401k match | Health savings account | Life insurance | Paid Holidays | Parental leaveSenior-level Full TimeEl Segundo, CA11h ago
-
Senior Analytics Engineer USD 147K-190KAgile | Airflow | CI/CD | DBT | Data GovernanceCasual environment | Company retreats | Learning from leaders | Wellness programSenior-level Full TimeNew York, New York11h ago
-
Data Analytics Integrator USD 92K-158KData Engineering | Data Modeling | Data Pipelines | Data Quality | Data Reconciliation401k | Dental insurance | Discretionary PTO | Health insurance | Professional development opportunitiesMid-level Full TimeAtlanta, GA12h ago
-
Senior Backend Engineer- AI Agents (Remote) USD 180K-240KA/B | A/B Testing | API Design | AWS | AzureFlexible vacation policy | Health insurance coverage | Team offsites and in person collaboration | Work with distributed teamSenior-level Full TimeUnited States R12h ago
-
Senior Data Engineer, Solutions Architecture USD 110K-145KAPI | AWS | Access Control | Airflow | AzureSenior-level Full TimeSan Francisco, California, United States13h ago
-
Senior Data Engineer, Solutions Architecture USD 110K-145KAPI Integration | AWS | Access Control | Airflow | AnthropicCompany-provided meals and events | Hybrid work environmentSenior-level Full TimeSan Diego, California, United States13h ago
-
Senior Data Engineer, Solutions Architecture USD 110K-145KAPI Integration | AWS | Access Control | Anthropic | AzureAutonomous role | Hybrid work environment | Opportunities for professional growthSenior-level Full TimeScottsdale, Arizona, United States13h ago
-
Software Engineer, Machine Learning USD 150K-215KAPI Design | AWS | AWS CloudWatch | Artificial Intelligence | AzurePartial telecommutingMid-level Full TimeBoston, MA13h ago
-
Senior Databricks Platform Engineer USD 180K-247KActive Directory | Apache Hive | Apache Spark | Auditing | AutoscalingPublic Trust DHS EODSenior-level Full TimeArlington, VA, United States15h ago
-
Mid-level Full TimeSioux Falls, SD, United States16h ago
-
Senior Data Engineer USD 135K-180KAKS | Automated testing | Azure Cosmos | Azure Cosmos DB | Azure DataSenior-level Full TimeAtlanta, GA | Kansas City, MO …16h ago
-
Apache Spark | Data Lineage | Data Modeling | Data Quality | Data Quality ManagementRemote work | W2 employmentMid-level Full TimeChicago, IL. R17h ago
-
Data Modeling, Fraud Analytics Consulting (Remote) USD 104K-174KAWS | Data Modeling | Data Science | Fraud Detection | Machine LearningDental insurance | Flexible time off | Flexible work environment | Hybrid work | Medical insuranceMid-level Full TimeUnited States, UNITED STATES, United States R19h ago
-
AI Engineer USD 141K-236KAzure OpenAI | Azure Security | Azure Security Center | Azure Synapse | DatabricksDisability insurance | Health insurance | Holiday pay | Learning and development | Life insuranceSenior-level Full TimeUSA-DC-Washington19h ago
-
Senior Machine Learning Engineer (Multiple Positions) USD 194K-355KApache Spark | Big Data | Data Preparation | Deep learning | Experiment designSenior-level Full TimeSeattle, Washington, United States20h ago
-
Senior Embedded Software Engineer - Flight Software USD 100K-200KBare Metal | Bash | C# | C++ | CANSenior-level Full TimeSaratoga, CA1d ago
-
Senior Machine Learning Engineer (Computer Vision & AI) USD 149K-198KAgile Development | Anomaly Detection | Azure | CUDA | Cloud ComputingTraining opportunitiesSenior-level Full TimeAUBURN HILLS HQ R&D, MI, US1d ago
-
CI/CD | Fastlane | Git | Multithreading | SOLIDAsynchronous culture | Fast-paced environment | Learning Support | Remote workSenior-level Full TimeKansas City, MO, USA1d ago
-
AI-Advanced Technician USD 99K-158KAWS | Azure | Data Modeling | Data Structures | Google Cloud401k matching | Paid parental leaveMid-level Full TimeTysons Corner, VA, United States1d ago