Model Serving Engineer
Tasks
- Build autoscaling and capacity management systems
- Collaborate with ML and product teams for model releases
- Create deployment workflows canary shadow testing rollback
- Design model serving platforms for production
- Develop end to end observability and error tracking
- Document operational procedures performance characteristics and tuning guidance
- Implement caching prompt deduplication response reuse
- Implement multi tenant routing rate limiting quality of service
- Implement security controls request signing content filtering abuse detection
- Integrate model serving with API gateways identity and observability
- Lead incident response for high availability AI services
- Optimize inference performance
- Tune GPU utilization memory management KV cache strategies
Perks/Benefits
Skills/Tech-stack
API Gateway | Abuse detection | Automated rollback | Autoscaling | C++ | Caching | Canary Releases | Capacity Planning | Content Filtering | Continuous batching | Distributed Systems | GPU Utilization | Go | Incident Response | Inference Optimization | KV cache | Kubernetes | LLM Inference | Memory Management | Metrics | Multi-tenant | Multi-tenant routing | Observability | Performance Engineering | Prompt Deduplication | Python | Quality of Service | Rate Limiting | Request Multiplexing | Request Signing | Rust | Security controls | Shadow testing | Structured Logging | Tenant Routing | TensorRT-LLM | Tracing | VLLM
Education
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R12d ago
-
Machine Learning Engineer / AI Model Developer (Mid to Senior)(Top Secret Clearance Required) (Hybrid) USD 78K-176KAPIs | Agile methodologies | Backend Development | Cloud Computing | Data Preparation401k | Employee discount program | Employee referral rewards | Flexible spending account | Flexible work scheduleMid-level Full TimeFort Belvoir, VA, US R6h ago
-
Sr. Embedded Software/Firmware Engineer USD 180K-240KARM | ASPICE | AUTOSAR | C# | C++Competitive compensation | Dental insurance | Flexible hours | Medical insurance | Pre-IPO stock optionsSenior-level Full TimeUnited States - Remote R17h ago
-
Senior AI Engineer, Tools & Agents USD 200K-240KAPI Design | Agent Orchestration | Knowledge Base | LLM | Multi-tenantDental insurance | Flexible PTO | Health insurance | Parental leave | Visa supportSenior-level Full TimeSan Francisco R19h ago
-
Sr. Engineer, Firmware & Embedded Software USD 79K-158KAgile | Azure DevOps | BDM | Bare Metal | BootloaderOccasional travel for onboarding | Remote work within the United StatesSenior-level Full TimeFlorida, United States R20h ago
-
Data Solutions Engineer - HYBRID/NC RESIDENTS ONLY USD 119K-164KApache Spark | Azure | Azure Data | Azure Data Factory | Azure Data Lake401k match | Dental insurance | Employee referral program | Flexible spending account | Health insuranceEntry-level Full TimeDurham, North Carolina, United States R20h ago
-
Senior Data Engineer - Digital Pathology (Hybrid) USD 98K-163KAPI Integration | Agile | Apache Airflow | Apache Spark | Batch ProcessingSenior-level Full TimeRochester, MN, United States R20h ago
-
Senior Software Engineer, MLOps USD 175K-220KAgile | Batch Processing | Consistency guarantees | Data Modeling | Data Pipelines401k match | Charitable donation match | Commuter benefits | Dental insurance | Flexible time offSenior-level Full TimeRemote - United States R21h ago
-
Senior Data Engineer - Finance USD 172K-203KARR | AWS | Apache Iceberg | Apache Spark | CRMDental insurance | Employee discounts | Flexible wellness time off | Health insurance | Hybrid workSenior-level Full TimeHybrid - San Francisco, California R22h ago
-
Senior-level Full TimeRemote - United States R22h ago
-
Sr. Data Engineer (HYBRID) USD 145K-160KAWS | AWS Glue | AWS Lambda | Amazon Athena | Amazon DynamoDBComplimentary club membership | Hybrid work schedule | Personal training | Pilates | Shop discountsSenior-level Full TimeNew York, NY, United States R22h ago
-
Staff Software Engineer, Data Products USD 171K-224KAWS | AWS Lambda | Amazon Athena | Amazon Redshift | Amazon S3401k retirement savings plan | Flexible time off | Generous parental leave | Health, dental, and vision insurance | Lifestyle spending account mental health support solutionsSenior-level Full TimeRemote, USA R23h ago
-
Senior Software Engineer (Data Engineer) USD 138K-162KAWS | Akka | Algorithms | Apache Spark | Cloud Native401k plan company match | Basic life insurance | Dental insurance | Health insurance | Vision insuranceSenior-level Full TimeUnited States R23h ago
-
Senior Data Engineer - Agentic AI Engineering USD 138K-173KAWS | Access Control | Airflow | Azure | Batch dataRemote-friendlySenior-level Full TimeUnited States of America R1d ago
-
AWS | AWS CDK | AWS CodeBuild | AWS CodePipeline | AWS Secrets401k | Healthcare coverage | PTO | Paid Company Holidays | Phone stipendSenior-level Full TimeSan Carlos - Hybrid R1d ago
-
Autonomy | C plus plus | CPU GPU | CPU/GPU Optimization | Critical softwareHealthcare | Paid Company Holidays | Paid time off | Phone stipend | Wellness benefitsSenior-level Full TimeSan Carlos - Hybrid R1d ago
-
Data Engineer USD 90K-151KAPIs | Amazon Redshift | CI/CD | Cloud Computing | Cloud platform401k match | Dental insurance | Equipment provided | Health insurance | PTOMid-level Full TimeRemote, US R1d ago
-
Senior Machine Learning Engineer, Safety USD 216K-303KData Preprocessing | Deep learning | Distributed Training | Embeddings | Hugging FaceSenior-level Full TimeRemote - United States R1d ago
-
Algorithms | Concurrency | Data Structures | Database systems | Distributed SystemsCompany-matched 401k | Flexible schedule | Health coverage | Parental leave | Remote work flexibilitySenior-level Full TimeUnited States R1d ago
-
Algorithms | Concurrent programming | Data Structures | Distributed Systems | ESQL401k match | Flexible schedule | Generous vacation days | Health coverage | Parental leaveSenior-level Full TimeUnited States R1d ago
-
Applied AI Engineer USD 254K-328KAgentic Workflows | Embeddings | Golden datasets | Knowledge graph | LLM401k | Bi annual in person offsites | Equity | Visa sponsorshipSenior-level Full TimeRemote, US R1d ago
-
Senior Solution Engineer, HCLS USD 165K-216KAnalytics | Cloud Computing | Data Lake | Data Warehouse | Data fabricTravel opportunitiesSenior-level Full TimeUS-IN-Remote R1d ago
-
Algorithms | Concurrency | Data Structures | Distributed Systems | Engine technologyCompany-matched 401k | Donation matching | Employee stock program | Flexible schedule | Health coverageSenior-level Full TimeUnited States R1d ago
-
Machine Learning Engineer (Remote) CAD 401KConcurrency | Data Pipelines | Data Processing | Data Structures | Distributed SystemsConference support | Health benefits | Mental health care | Paid time off | Parental leaveMid-level Full TimeAlberta, Canada; British Columbia; Calgary, Alberta, … R1d ago
-
Senior AI Engineer, Contract To Hire USD 130K-195KAWS | Agentic AI | Cloud Native | Cloud platform | Cloud-native developmentSenior-level Full TimeRemote, United States R1d ago