Senior Data Engineer - Streaming
US - GA - Atlanta, United States
USD 152K-178K Senior-level Full Time
Tasks
- Build event driven ingestion into Microsoft Fabric
- Configure dead letter queue routing
- Design latency blueprints for SLAs
- Design streaming architecture playbook
- Handle schema drift with metadata driven frameworks
- Implement exactly once and idempotent processing
- Implement replay and backfill mechanisms
- Manage checkpoint and state
- Monitor stream health metrics
- Template stateful stream processing concepts
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Flink | Apache Kafka | Avro | Azure Event | Azure Event Hubs | Checkpointing | Dead Letter | Dead Letter Queue | Delta Lake | Event Hubs | Exactly once | Idempotency | JSON Schema | JSON Schema Registry | KQL | Kafka Streams | Microsoft Fabric | PySpark | Scala | Schema Registry | Spark Streaming | Spark Structured Streaming | Structured Streaming | Watermarking | Windowing
Education
N/A
Roles
Related jobs
-
Mid-level Full TimeNew Jersey, New Jersey, United States1d ago
-
Sr. Data Engineer USD 61K-94KApache Flink | Apache Kafka | Apache NiFi | Apache Spark | Cassandra401k match | Accidental death and dismemberment insurance | Bonus plan | Commuter benefits | Flexible spending accountsSenior-level Full TimeCordova, TN, United States1d ago
-
Senior-level Full TimeUnited States1d ago
-
Senior Principal Software Engineer, Machine Learning USD 222K-453KA/B | A/B Testing | B testing | Canary Deployment | Capacity PlanningHybrid work model | MentorshipSenior-level Full TimeRemote, US R2d ago
-
A/B | A/B Testing | B testing | Distributed Training | Embeddings401k matching | Commuter benefits | Dental insurance | FSA | HSASenior-level Full TimeMountain View, California, United States2d ago
-
A/B | A/B Testing | Auction theory | B testing | Constrained optimization401k employer match | Disability benefits | Family-forming benefits | Flexible spending account | Health insuranceSenior-level Full TimeUSA - Remote, United States R2d ago
-
Senior Software Engineer - Machine Learning USD 183K-223KApache Flink | Apache Spark | Apache Storm | Data Modeling | Deep learning401k match | Dental insurance | Health insurance | Life insurance | Long-term disabilitySenior-level Full TimeSan Francisco, CA 94158, United States R2d ago
-
Senior Software Engineer, Machine Learning Infrastructure (Tinder LLC, West Hollywood, California) USD 190K-246KA/B | A/B Testing | AWS | Amazon ECS | Amazon EKSDomestic travel up to 10 percent | On site meetings and trainings | Telecommuting permittedSenior-level Full TimeWest Hollywood, California2d ago
-
Sr. Software Engineer, Data Products USD 105K-175KAPI Design | Apache Airflow | Asynchronous processing | Backend Development | Backward Compatibility401k retirement savings plan | Adoption Assistance | Charitable gift program | Company holidays | Dental insuranceSenior-level Full TimeChicago, Illinois, United States R2d ago
-
Data Engineer III/Senior (Analytics) USD 130K-170KCI/CD | DORA Metrics | Data Governance | Data Modeling | Data Privacy401k retirement savings plan | Dental insurance | Employee stock purchase plan | Health insurance | Life insuranceSenior-level Full TimeEdison, NJ, United States R2d ago
-
Data Engineer III/Senior (Analytics) USD 130K-170KAnalytics engineering | Azure | CCPA | CI/CD | DORA Metrics401k retirement plan | Dental insurance | Employee stock purchase plan | Health insurance | Life insuranceSenior-level Full TimeIrving, TX, United States R2d ago
-
Data Engineer III/Senior (Analytics) USD 130K-170KAutomation | Azure | CI/CD | DORA Metrics | Data Governance401k retirement plan | Dental insurance | Employee stock purchase plan | Health insurance | Life insuranceSenior-level Full TimeEllenwood, GA, United States R2d ago
-
Data Engineer III/Senior (Analytics) USD 130K-170KAnalytics engineering | Azure | CCPA | CI/CD | DORA Metrics401k matching | Dental insurance | Employee stock purchase plan | Health insurance | Life insuranceSenior-level Full TimeRomulus, MI, United States R2d ago
-
API Design | AWS | Agent systems | Airflow | Apache IcebergBackup childcare | Financial coaching | Health care coverage | Mental health support | On-site health and wellness centersSenior-level Full TimeHouston, TX, United States2d ago
-
Senior Software Engineer, Machine Learning Infrastructure (Tinder LLC, West Hollywood, California) USD 190K-246KAWS | Amazon ECS | Amazon EKS | Apache Flink | Apache KafkaDomestic travel for team meetings and on site trainings | Telecommuting permittedSenior-level Full TimeWest Hollywood, California2d ago
-
Data Scientist, AI/ML Model Quality USD 143K-192KApache Superset | Bayesian Inference | Causal Graphs | Confidence Calibration | Data DriftMid-level Full TimeAustin; New York City; San Diego2d ago
-
Sr. Databricks Consultant - Top Secret Clearance USD 175K-250KAWS S3 | Apache Spark | Azure Data | Azure Data Lake | CI/CDTop Secret clearanceSenior-level Full TimeFAIRFAX, VA, United States2d ago
-
Sr. Databricks Consultant - TS/SCI-FSP USD 175K-250KAWS | AWS S3 | Apache Spark | Azure | Azure DataAdditional funding contingent | Benefits package | Travel as neededSenior-level Full TimeFAIRFAX, VA, United States2d ago
-
Sr. Databricks Consultant - DHS Public Trust USD 175K-250KAWS S3 | Apache Spark | Azure Data | Azure Data Lake | CI/CDSenior-level Full TimeWork from home, VA, United States2d ago
-
Hiring for Senior AI Software Engineer (Agentic AI / AI Agents) | Hybrid | W2 Role | F2F Interview in NJ USD 115K-150KAI Agents | AWS | Agentic AI | Audit Logging | AzureHybrid work | In-person interviewSenior-level Contract Full TimeJersey City, NJ, United States2d ago
-
Principal Data Engineer – (Hadoop/Big Data, AWS, Python, Kinesis) - Irvine, CA - Onsite - Contract to Hire - AS A USD 132K-165KAmazon Kinesis | Amazon RDS | Amazon Redshift | Amazon Web Services | Apache HBaseSenior-level Full TimeIrvine, CA2d ago
-
AI-assisted software development | API Design | AWS | Airflow | Apache IcebergBackup childcare | Financial coaching | Health care coverage | Mental health support | On-site health and wellness centersSenior-level Full TimeHouston, TX, United States2d ago
-
Senior Data Engineer ID75059 USD 155K-170KAPI Integration | AWS | Apache Airflow | Apache Spark | AvroEducation budget | Exciting projects | Fitness budget | Flexible schedule | MentorshipSenior-level Full TimeBaltimore, United States R2d ago
-
Senior Data Engineer ID75059 USD 155K-170KAPI Integration | AWS | Apache Airflow | Apache Spark | AvroEducation budget | Exciting projects | Fitness budget | Flexible schedule | MentorshipSenior-level Full TimeHouston, United States R2d ago
-
Senior Data Engineer ID75059 USD 155K-170KAPI Integration | AWS | Airflow | Apache Kafka | AvroFlexible schedule | Mentorship | Personalized growth roadmaps | Professional growth | Remote and office optionsSenior-level Full TimeWest Palm Beach, United States R2d ago