Senior Engineer, Inference Data Plane
Tasks
- Architect and refine system designs for multi tenant high scale inference cloud
- Build Kubernetes native distributed inference frameworks and routing
- Contribute upstream to inference serving open source projects
- Implement and optimize distributed inference hosting
- Implement flow control and load balancing for LLM serving
- Lead end to end design development and delivery of data plane services for AI inference
- Maintain and operate critical high scale services with observability and SLOs
- Mentor and coach junior engineers
- Optimize KV cache locality and reduce tail latency
Perks/Benefits
- Employee assistance program
- Employee stock purchase program
- Equity compensation
- Flexible time off
- LinkedIn Learning access
- Local Employee Meetups
- Reimbursement for conferences training and education
- Remote work
Skills/Tech-stack
Continuous batching | Data parallelism | Distributed Systems | GRPC | Go | KServe | KV cache | Kubernetes | LLM serving | Llm-d | NIXL | Nvidia Dynamo | Paged Attention | Prefix caching | Python | Quantization | Ray Serve | SGLang | Tensor Parallelism | TensorRT | TensorRT-LLM | VLLM
Education
N/A
Related jobs
-
Azure Data | Azure Data Factory | Azure DevOps | Azure Machine Learning | CI/CDRemote | USC OnlySenior-level Full TimeDallas, TX. R3h ago
-
ABAC | AWS CloudFormation | AWS Lambda | AWS Step Functions | AWS-APIsOnsite/hybrid workSenior-level Full TimeChantilly, Virginia R3h ago
-
AI Developer Builder USD 140K-180KAWS | Agentic Systems | Anthropic Bedrock | Context pipelines | DatadogDirect input on roadmap | High ownershipSenior-level Full TimeTel Aviv-Yafo, Tel Aviv District, IL R6h ago
-
(US) Principal ML System Engineer USD 139K-304KAWS | Access Control | Alerting | Audit Logging | AuthenticationSenior-level Full TimeRemote, USA R14h ago
-
Senior Machine Learning Engineer, Ads ML Platform USD 216K-303KAnomaly Detection | Data Governance | Data Infrastructure | Data Lineage | Data Storage401k employer match | Caregiving support | Comprehensive healthcare benefits | Family planning support | Flexible remote workSenior-level Full TimeRemote - United States R18h ago
-
Sr. Platform Engineer, Kubernetes USD 117K-200KAWS EKS | Alluxio | Amazon S3 | Ansible | Apache FlyteEmployee benefits | Onsite collaboration flexible remote optionSenior-level Full TimePA - West Chester, 1354 Boot … R19h ago
-
Senior Software Engineer - Storage Control Plane USD 266K-395KBlock Storage | C# | C++ | CI/CD | CephSenior-level Full TimeSan Francisco Office (Fremont St) R19h ago
-
A/B | A/B Testing | B testing | Data parallelism | Deep learning401k employer match | Family planning support | Flexible vacation | Gender-affirming care | Global benefit programsSenior-level Full TimeRemote - United States R20h ago
-
A/B | A/B Testing | B testing | C++ | Cloud Computing401k employer match | Family planning support | Flexible vacation | Gender-affirming care | Global benefitsSenior-level Full TimeRemote - United States R20h ago
-
Applied AI/ML Engineer USD 150K-200KAPI Integration | Anomaly Detection | Apache Spark | CI/CD | Cloud Platforms401k match | Comprehensive benefits package | Flexible PTO | Paid Holidays | Remote workMid-level Full TimeWashington DC area R21h ago
-
Finance & Strategy AI Engineer USD 146K-246KAPI Integration | Adaptive Planning | Agentic Workflows | BigQuery | Data GovernanceFlexible remote work | Health benefits | Parental leave | Professional development stipendSenior-level Full TimeSan Francisco - SF9 R21h ago
-
AI Assisted Query Profiling | AI assisted code | AI-assisted code review | ClickHouse | Code reviewSenior-level Full TimeUnited States R21h ago
-
Software Engineer, Data Infrastructure - AI Platform USD 200K-230KAI-assisted tooling | ClickHouse | Data Pipelines | Debugging | Distributed SystemsSenior-level Full TimeUnited States R21h ago
-
Staff Data Platform Engineer - AI Platform USD 200K-230KClickHouse | Compliance | Data Pipelines | Debugging | Distributed SystemsSenior-level Full TimeUnited States R21h ago
-
Senior Data Platform Engineer - AI Platform USD 200K-230KAI code review | AI-assisted debugging | ClickHouse | Code review | Data Pipeline ReliabilitySenior-level Full TimeUnited States R21h ago
-
Data Platform Engineer - AI Platform USD 200K-230KClickHouse | Data Pipelines | Distributed Systems | Incident Response | Log AnalysisSenior-level Full TimeUnited States R21h ago
-
Senior DevOps Engineer (Storage) - remote in the US USD 200K-290KAir-gapped | ArgoCD | CAPI | CSI | Cluster APICompany outings | Conference attendance | Hackathons | Happy hours | Professional development and trainingExecutive-level Full TimeRemote, USA, United States R22h ago
-
Senior Software Engineer (Storage) - remote in the US USD 100K-170KArgoCD | CSI | Cluster API | Flux | GitOpsCompany outings and happy hours | Conferences and working groups | Hackathons and tech talks | Professional development and trainingSenior-level Full TimeRemote, USA, United States R22h ago
-
Autonomy Algorithms Software Engineer USD 115K-145KAgent systems | Agile | Algorithms | C++ | CI/CDHybrid work environment | Remote work optionMid-level Full TimeArlington, VA R22h ago
-
Autonomy Algorithms Software Engineer USD 115K-145KAgent systems | Agile | Algorithms | C++ | CI/CDFlexible work schedule | Hybrid work environment | Remote work availableMid-level Full TimeWoburn, MA R22h ago
-
Forward Deployed Engineer USD 280K-320KAgent Development | Deployment | Evaluation | JavaScript | Language ModelsFlexible working hours | Generous vacation | Parental leave | Visa sponsorshipMid-level Full TimeNew York City, NY; San Francisco, … R22h ago
-
Software Engineer, Embedded Agentic AI USD 170K-210KC# | C++ | Cloud Platforms | Containerization | DockerDisability benefits | Equity awards | Health insurance | Life insurance | Paid time offEntry-level Full TimeSan Jose, California R23h ago
-
Senior Software Engineer, Data Platform USD 196K-245KAirflow | BigQuery | Cost Optimization | Dagster | Data GovernanceBenefits | Equity | Relocation assistanceSenior-level Full TimeSan Francisco Bay Area or Remote … R23h ago
-
Senior Research Engineer, LLM Training & Post-Training USD 165K-310KCUDA | DPO | DeepSpeed | Distributed Training | Experiment design401k matching | Company holidays | Comprehensive health coverage | Dental coverage | Flexible time offSenior-level Full TimeNew York, New York, United States; … R23h ago
-
Staff Data Engineer - AI Platform USD 200K-230KAI Assisted Query Profiling | AI assisted code | AI-assisted code review | ClickHouse | Code reviewSenior-level Full TimeUnited States R23h ago