Senior Software Engineer - Real-Time Ingestion
Tasks
- Build Kafka based streaming infrastructure
- Collaborate on Cloud Spanner schema and write patterns
- Create pipeline documentation and runbooks
- Design monitoring and data quality checks
- Develop real-time streaming pipelines
- Ensure privacy compliant handling of sensitive data
- Implement Apache Beam Dataflow jobs
- Implement dead letter queues and retry logic
- Optimize pipeline performance and reduce lag
- Participate in on call rotation for uptime SLA
- Partner with data producers on event schemas and quality
- Troubleshoot production pipeline failures and data issues
Perks/Benefits
Skills/Tech-stack
APICurio | Amazon DynamoDB | Amazon Kinesis | Apache Beam | Apache Dataflow | Apache Kafka | Apache Spark | Apache Spark Streaming | Avro | Avro serialization | CCPA | Cloud Spanner | Cloud Storage | Confluent Schema Registry | Exactly once | GDPR | Google BigQuery | Google Cloud | Google Cloud Storage | Google Pubsub | JSON | Java | Kafka Schema Registry | PII protection | Parquet | Protobuf | Protobuf serialization | Python | SQL | Scala | Schema Registry | Spark Streaming | State management | Watermarks | Windowing
Education
Roles
Related jobs
-
Machine Learning Co-Op (Fall 2026) USD 84K-176KDAX | Deep learning | Git | GitHub | Machine LearningMid-level Full TimeCanton, OH, United States1h ago
-
Machine Learning Co-Op (Summer 2026) USD 96K-180KDAX | Data Analysis | Data Visualization | Deep learning | GitMid-level Full TimeCanton, OH, United States1h ago
-
Senior-level Full TimeNew York, NY, United States1h ago
-
Senior-level Full TimeNew York, NY, United States1h ago
-
Manager of Data Engineering USD 140K-160KARM Templates | Access Control | Active Directory | Apache Kafka | Apache Spark401k match | Accidental death and dismemberment insurance | Certification support | Dental insurance | Employee assistance programMid-level Full TimeVancouver, WA, United States2h ago
-
Senior MLOps/AI Engineer USD 165K-185KAPI Development | AgenticAI Ops | Automation | Containerization | Data Pipelines401k matching | Dental insurance | Disability insurance | Employee stock purchase plan | FSASenior-level Full TimeWork From Home, United States R2h ago
-
AWS | AWS DMS | AWS Glue | AWS IAM | AWS LambdaJira time trackingMid-level ContractCarson City, United States5h ago
-
Amazon Web Services | Analysis Services | Data Modeling | Database Design | ETLHybrid work environmentSenior-level ContractNashville, United States5h ago
-
Sr. Gen AI Developer USD 100KAI Agents | AI Governance | AI copilots | Auditability | Bias MitigationSenior-level Full TimeUS-FL-Tampa6h ago
-
Senior-level Full TimeUS-NJ-Jersey City6h ago
-
Senior-level Full TimeUS-TX-Irving6h ago
-
Senior ML Platform Engineer - Artificial Intelligence USD 160K-240KArgo | Buildpacks | CI/CD | Container Orchestration | Data Structures401k match | Dental insurance | Life insurance | Medical insurance | Paid HolidaysSenior-level Full TimeNew York6h ago
-
Behavioral analytics | Data Analysis | Data Warehousing | Data correlation | Deep learningSenior-level Full TimeSan Jose, California, United States6h ago
-
Algorithm Development | Behavioral Data | Behavioral Data Analysis | Data Analysis | Data WarehousingSenior-level Full TimeSan Jose, California, United States6h ago
-
Data Engineer, Analytics USD 191K-235KA/B | A/B Testing | B testing | Big Data | ClusteringTelecommuting allowedSenior-level Full TimeMenlo Park, CA | Remote, US R7h ago
-
Data Engineer, Analytics USD 209K-235KBig Data | Data Governance | Data Modeling | Data Quality | Data SecurityTelecommutingSenior-level Full TimeMenlo Park, CA | Remote, US R7h ago
-
Data Engineer, Analytics USD 205K-235KBig Data | Data Governance | Data Modeling | Data Quality | Data SecurityTelecommutingEntry-level Full TimeMenlo Park, CA | Remote, US R7h ago
-
Data Scientist, Analytics USD 193K-196KA/B | A/B Testing | B testing | Clustering | Descriptive StatisticsMid-level Full TimeBellevue, WA7h ago
-
Data Engineer, Analytics USD 185K-200KAmazon EC2 | Amazon Redshift | Amazon S3 | Apache Druid | Apache SparkTelecommuting from anywhere in the United States allowedMid-level Full TimeMenlo Park, CA | Remote, US R7h ago
-
Data Engineer USD 183K-196KBusiness Intelligence | Data Governance | Data Instrumentation | Data Modeling | Data PrivacyMid-level Full TimeMenlo Park, CA7h ago
-
Data Engineer USD 185K-196KCSS | Data Modeling | Data Warehousing | Dimensional data | Dimensional data modelingMid-level Full TimeMenlo Park, CA7h ago
-
Benchmarking | Code review | Machine Learning | PyTorch | PythonSenior-level Full TimeNew York, NY7h ago
-
Staff Software Engineer, Cloud Platforms Infrastructure USD 207K-300KC++ | Cloud Storage | Cloud storage systems | Data Structures | Data Structures and AlgorithmsSenior-level Full TimeSunnyvale, CA, USA7h ago
-
Full Stack Tech Lead, Search Experiments USD 207K-300KDistributed Systems | Google Infrastructure | Java | Kotlin | Search infrastructureSenior-level Full TimeCambridge, MA, USA7h ago
-
Machine Learning Engineer, Computer Vision USD 174K-252KC++ | Computer Vision | Data Processing | Debugging | Deep learningMid-level Full TimeMountain View, CA, USA7h ago