Senior Engineer, Inference Data Plane
Tasks
- Architect and refine system designs for multi tenant high scale inference cloud
- Build Kubernetes native distributed inference frameworks and routing
- Contribute upstream to inference serving open source projects
- Implement and optimize distributed inference hosting
- Implement flow control and load balancing for LLM serving
- Lead end to end design development and delivery of data plane services for AI inference
- Maintain and operate critical high scale services with observability and SLOs
- Mentor and coach junior engineers
- Optimize KV cache locality and reduce tail latency
Perks/Benefits
- Employee assistance program
- Employee stock purchase program
- Equity compensation
- Flexible time off
- LinkedIn Learning access
- Local Employee Meetups
- Reimbursement for conferences training and education
- Remote work
Skills/Tech-stack
Continuous batching | Data parallelism | Distributed Systems | GRPC | Go | KServe | KV cache | Kubernetes | LLM serving | Llm-d | NIXL | Nvidia Dynamo | Paged Attention | Prefix caching | Python | Quantization | Ray Serve | SGLang | Tensor Parallelism | TensorRT | TensorRT-LLM | VLLM
Education
N/A
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R14d ago
-
Explosives Research Experiment Lead USD 146K-222KAerospace Engineering | Bench testing | C++ | Computational modeling | Computer Aided Design401k | Education reimbursement | Flexible schedule | Hybrid work schedule | Relocation assistanceSenior-level Full TimeLivermore, CA, United States R11h ago
-
Machine Learning Engineer USD 80K-90KDeep learning | Evaluation metrics | Generalization | Language Models | Large Language ModelsSenior-level Full TimeFremont, California R12h ago
-
Senior Software Engineer (Storage) - remote in the US USD 100K-150KArgo CD | ArgoCD | Block Storage | CSI | Cluster APISenior-level Full TimeBoston, MA, United States R13h ago
-
Applied AI Lead USD 140K-180KAPI Integration | Agents | Artificial Intelligence | Assistants | Context engineeringEquity package | Health coverage | Remote-first workplace | Unlimited PTOSenior-level Full TimeRemote - United States or Canada R13h ago
-
Principal Applied AI Engineer USD 185K-245KAgent Frameworks | Agent SDK | Anthropic Agent SDK | Context engineering | EvalsSenior-level Full TimeRedesign Health R13h ago
-
AI Forward Deployed Engineer | $120K-$150K + Hybrid + Equity | AI-Based Outage Intelligence Company A USD 120K-150KAI workflows | APIs | Automation | Backend Development | IntegrationDirect impact on company growth | Equity | Hybrid work | Significant ownership | Unlimited PTOMid-level Full TimeKing of Prussia, PA, United States R13h ago
-
API Design | Agentic AI | Cloud Computing | Data Infrastructure | Deployment401k match | Career advancement | Dental insurance | Flexible time off policy | Life insuranceSenior-level Full TimeDallas, Texas, United States; United States R14h ago
-
Machine Learning Research Engineer USD 200K-330KAWS | Azure | Benchmarking | CUDA | DDP401k match | Dental insurance | Health insurance | Paid time off | Vision insuranceMid-level Full TimeEmeryville, California, United States; Hybrid (2-3 … R16h ago
-
Staff Engineer, Inference Optimizations USD 191K-239KBF16 | Bandwidth Optimization | CUDA | Expert Routing | FP4Conference reimbursement | Education reimbursement | Employee assistance program | Flexible time off | LinkedIn Learning accessSenior-level Full TimeDenver R16h ago
-
Staff Engineer, Inference Optimizations USD 191K-239KAttention Mechanisms | BF16 | CUDA | CUDA kernels | Distributed SystemsConference reimbursement | Education reimbursement | Employee assistance program | Flexible time off | LinkedIn LearningSenior-level Full TimeBoston R16h ago
-
Staff Engineer, Inference Optimizations USD 191K-239KAMD GPU | Attention Optimization | Bandwidth Optimization | Benchmarking | CUDAConference reimbursement | Employee assistance program | Flexible time off | LinkedIn Learning access | Local Employee MeetupsSenior-level Full TimeAustin R16h ago
-
Staff Engineer, Inference Optimizations USD 191K-239KAttention Optimization | BF16 | CUDA | FP4 | FP8Conference reimbursement | Education reimbursement | Employee assistance program | Employee stock purchase program | Equity compensationSenior-level Full TimeSan Francisco R16h ago
-
Machine Learning Engineer USD 155K-228KAWS | Airflow | Argo CD | Azure | CI/CDGenerous time off | Healthcare | Paid parental leave | Paid personal time off | Paid sick timeMid-level Full TimeRemote - US R16h ago
-
Senior Software Engineer I - AI Inference Data Plane USD 139K-174KAutoscaling | Continuous batching | Data parallelism | Distributed Systems | GRPCConference reimbursement | Education reimbursement | Employee assistance program | Employee stock purchase program | Equity compensationSenior-level Full TimeAustin R17h ago
-
Senior Engineer, Inference Data Plane USD 139K-174KContinuous batching | Data parallelism | Distributed Systems | GRPC | GoConference reimbursement | Employee assistance program | Employee stock purchase program | Equity compensation | Flexible time offSenior-level Full TimeBoston R17h ago
-
Senior Software Engineer I - AI Inference Data Plane USD 139K-174KAutoscaling | Continuous batching | Data parallelism | Distributed Systems | GRPCConference reimbursement | Employee assistance program | Employee stock purchase program | Equity compensation | Flexible time offSenior-level Full TimeSan Francisco R17h ago
-
Senior Data Engineer (Databricks) USD 180K-248KAI tools | Agile | Apache Spark | Azure Data | Azure Data Factory401k match | Employee awards | Industry certifications support | Recognition programs | Remote or hybrid workSenior-level Full TimeUnited States R17h ago
-
APIs | Artificial Intelligence | Data Engineering | Language Processing | Machine LearningEquity compensation | Health benefitsSenior-level Full TimeSan Mateo, CA, United States R17h ago
-
Sr. Forward Deployed Engineer (FDE) - Retail USD 182K-250KAWS | Apache Spark | Azure | CI/CD | Data EngineeringCustomer-facing role | Travel 20 percent of the timeSenior-level Full TimeRemote - California R17h ago
-
Staff Software Engineer - AI Data Engine USD 172K-229KAWS | Agent systems | Agentic AI | Airflow | C++401k match | Dental insurance | Health savings account | Hybrid work | Life insuranceSenior-level Full TimeBoston, Massachusetts, United States; Las Vegas, … R18h ago
-
Data Scientist II, ML Infrastructure USD 114K-235KApache Airflow | Causal Inference | Code review | Distributed Computing | Feature importanceEntry-level Full TimePalo Alto, CA, US; Remote, US R18h ago
-
Senior Software Engineer, Data Layer USD 186K-218KAPI Platform | Caching | Distributed Systems | Go | GraphQL401k | Dental insurance | Medical insurance | Quarterly in person surges | Remote-firstSenior-level Full TimeRemote - USA R19h ago
-
Senior Staff Software Engineer, Data USD 235K-300KAccess Control | Amazon Redshift | Apache Airflow | BigQuery | Compliance401k employer match | Cell phone reimbursement | Competitive PTO | Corporate wellness program | Employee resource groupsSenior-level Full TimeUSA Remote R19h ago
-
Mid-Level Data Engineer USD 90K-98KAPI Development | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageContract award contingent | Remote workMid-level Full TimeWork from home, VA, United States R20h ago