Senior Software Engineer - Real-Time Ingestion
Tasks
- Build Kafka based streaming infrastructure
- Collaborate on Cloud Spanner schema and write patterns
- Create pipeline documentation and runbooks
- Design monitoring and data quality checks
- Develop real-time streaming pipelines
- Ensure privacy compliant handling of sensitive data
- Implement Apache Beam Dataflow jobs
- Implement dead letter queues and retry logic
- Optimize pipeline performance and reduce lag
- Participate in on call rotation for uptime SLA
- Partner with data producers on event schemas and quality
- Troubleshoot production pipeline failures and data issues
Perks/Benefits
Skills/Tech-stack
APICurio | Amazon DynamoDB | Amazon Kinesis | Apache Beam | Apache Dataflow | Apache Kafka | Apache Spark | Apache Spark Streaming | Avro | Avro serialization | CCPA | Cloud Spanner | Cloud Storage | Confluent Schema Registry | Exactly once | GDPR | Google BigQuery | Google Cloud | Google Cloud Storage | Google Pubsub | JSON | Java | Kafka Schema Registry | PII protection | Parquet | Protobuf | Protobuf serialization | Python | SQL | Scala | Schema Registry | Spark Streaming | State management | Watermarks | Windowing
Education
Roles
Related jobs
-
Machine Learning Co-Op (Fall 2026) USD 84K-176KDAX | Deep learning | Git | GitHub | Machine LearningMid-level Full TimeCanton, OH, United States3h ago
-
Machine Learning Co-Op (Summer 2026) USD 96K-180KDAX | Data Analysis | Data Visualization | Deep learning | GitMid-level Full TimeCanton, OH, United States3h ago
-
Sr. Platform Software Engineer – AI and Innovation USD 119K-180KAgile | Apollo GraphQL | Continuous Delivery | Fault-tolerant | Fault-tolerant systemsSenior-level Full TimeOak Brook, IL, United States3h ago
-
Mid-level Full TimeNew York, NY, United States3h ago
-
Senior-level Full TimeFlorham Park, NJ, United States3h ago
-
Senior Lead Architect USD 120K-205KAWS Lambda | Amazon Web Services | Bash | Bitbucket | C#Career growth | Inclusive work environment | Philanthropic foundation support | Professional development | Team collaborationSenior-level Full TimeAlpharetta, GA, United States3h ago
-
Senior Lead Software Engineers USD 120K-192KApex | Change Sets | Copado | Enterprise edition | GearsetSenior-level Full TimeAlpharetta, GA, United States3h ago
-
Senior Software Engineers USD 139K-257KAWS Lambda | Amazon Web Services | AppSync | Bitbucket | GitSenior-level Full TimeRoseland, NJ, United States R3h ago
-
Senior-level Full TimeNew York, NY, United States3h ago
-
Senior-level Full TimeNew York, NY, United States3h ago
-
Manager of Data Engineering USD 140K-160KARM Templates | Access Control | Active Directory | Apache Kafka | Apache Spark401k match | Accidental death and dismemberment insurance | Certification support | Dental insurance | Employee assistance programMid-level Full TimeVancouver, WA, United States4h ago
-
Senior MLOps/AI Engineer USD 165K-185KAPI Development | AgenticAI Ops | Automation | Containerization | Data Pipelines401k matching | Dental insurance | Disability insurance | Employee stock purchase plan | FSASenior-level Full TimeWork From Home, United States R4h ago
-
Python Engineer with Data Focus (Remote) USD 100K-130KAutomated testing | Azure DevOps | CI/CD | Databricks | DjangoFull-time | Fully remote | Long-term | No timezone shiftingSenior-level Full TimeFlorida, Aventura, United States of America R6h ago
-
AWS | AWS DMS | AWS Glue | AWS IAM | AWS LambdaJira time trackingMid-level ContractCarson City, United States7h ago
-
Amazon Web Services | Analysis Services | Data Modeling | Database Design | ETLHybrid work environmentSenior-level ContractNashville, United States7h ago
-
Sr. Gen AI Developer USD 100KAI Agents | AI Governance | AI copilots | Auditability | Bias MitigationSenior-level Full TimeUS-FL-Tampa7h ago
-
Senior-level Full TimeUS-NJ-Jersey City7h ago
-
Senior-level Full TimeUS-TX-Irving7h ago
-
Senior ML Platform Engineer - Artificial Intelligence USD 160K-240KArgo | Buildpacks | CI/CD | Container Orchestration | Data Structures401k match | Dental insurance | Life insurance | Medical insurance | Paid HolidaysSenior-level Full TimeNew York7h ago
-
Behavioral analytics | Data Analysis | Data Warehousing | Data correlation | Deep learningSenior-level Full TimeSan Jose, California, United States8h ago
-
Algorithm Development | Behavioral Data | Behavioral Data Analysis | Data Analysis | Data WarehousingSenior-level Full TimeSan Jose, California, United States8h ago
-
Data Engineer, Analytics USD 191K-235KA/B | A/B Testing | B testing | Big Data | ClusteringTelecommuting allowedSenior-level Full TimeMenlo Park, CA | Remote, US R9h ago
-
Data Engineer, Analytics USD 209K-235KBig Data | Data Governance | Data Modeling | Data Quality | Data SecurityTelecommutingSenior-level Full TimeMenlo Park, CA | Remote, US R9h ago
-
Data Engineer, Analytics USD 205K-235KBig Data | Data Governance | Data Modeling | Data Quality | Data SecurityTelecommutingEntry-level Full TimeMenlo Park, CA | Remote, US R9h ago
-
Data Scientist, Analytics USD 193K-196KA/B | A/B Testing | B testing | Clustering | Descriptive StatisticsMid-level Full TimeBellevue, WA9h ago