Applied Scientist, Agent Evaluation & Adaptive Model Routing
Singapore, SG / Austin, TX
USD 153K-250K (estimate) Mid-level Full Time
Tasks
- Build evaluation and decision systems
- Collaborate on billing access control and SLA engineering
- Conduct offline evaluation
- Design escalation and recovery policies
- Develop representative task suites
- Implement uncertainty aware selection
- Integrate evaluation and routing into production gateways
- Measure trajectory level behavior
- Own LLM and agent evaluation pipeline
- Research and prototype adaptive model routing strategies
- Run traffic replay and shadow testing
- Support internal pilots
Perks/Benefits
Skills/Tech-stack
A I | A I Inference | A/B | A/B Testing | Agent systems | B testing | Cascading | Contextual Bandits | Cost Latency Tradeoff | Data evaluation | Dataset Construction | LLM Evaluation | Language Processing | Machine Learning | Model Monitoring | Model routing | Natural Language | Natural Language Processing | Online Learning | Preference Modeling | Production Model Monitoring | PyTorch | Python | Regression testing | Shadow testing | Token Usage Analysis | Token usage | Traffic replay | Uncertainty Estimation | Uncertainty calibration | Usage analysis
Education
Regions
Countries
States
Cities
Related jobs
-
A/AI Research Engineer Asc - E1 USD 67K-124KC++ | CIN? N/A | Computer Vision | Control Engineering | Control Systems401k match | Dental insurance | Education assistance | Flexible work arrangements | Life insuranceEntry-level Full TimeOrlando, FL, US4h ago
-
Data Engineer Senior - Level 3 USD 85K-158KAgile | Data Cleansing | Data Mining | Data Quality | Data Warehousing401k match | Education assistance | Employee assistance program | Flexible work arrangements and schedules | HolidaysSenior-level Full TimeFort Worth, TX, US4h ago
-
Data Analyst Senior - Level 3 USD 85K-158KAutomation | C++ | Cause analysis | Chain management | Data Engineering401k match | Dental insurance | EAP | Education assistance | Flexible spending accountsSenior-level Full TimeFort Worth, TX, US4h ago
-
Agentic AI Engineer, Automation USD 220K-292KAgent Orchestration | C++ | Data Aggregation | Data Pipelines | Data SanitizationNone Full TimeCosta Mesa, California, United States8h ago
-
Senior Data Scientist USD 141K-165KBayesian Modeling | Causal Inference | Experimental Design | Forecasting | Geo-experiments401k matching | Annual class pass credits | Dental insurance | Extended holiday break | Flexible schedulesSenior-level Full TimeRemote, United States R8h ago
-
Data Scientist - Mid - #918 USD 100K-150KConfluence | Data Governance | Data Visualization | Data analytics | Data cleaning100% onsite | TS SCI clearance supportMid-level Full TimeDayton, Ohio, United States9h ago
-
Data Scientist - Senior - #917 USD 132K-189KArtificial Intelligence | Cloud infrastructure | Data Engineering | Data Governance | Data PipelinesSenior-level Full TimeDayton, Ohio, United States9h ago
-
Data Scientist - Expert - #914 USD 130K-193KAdvanced Analytics | Artificial Intelligence | Cloud Computing | Containerization | Data Engineering100 percent onsite | Security clearance TS SCISenior-level Full TimeDayton, Ohio, United States9h ago
-
API Integration | Artificial Intelligence | Data Engineering | Data Pipelines | Data provenanceSenior-level Full TimeUS026 PA Spring House - 1400 …9h ago
-
Senior Data Scientist, Biologics Discovery USD 109K-174KCloud Computing | Data Leakage Risk | Data leakage | Dataset curation | Distribution ShiftSenior-level Full TimeUS026 PA Spring House - 1400 …9h ago
-
Machine Learning Engineer, Relevance and Personalization USD 166K-195KA/B | A/B Testing | Apache Airflow | Apache Kafka | Apache SparkMid-level Full TimeUnited States R10h ago
-
Data Engineer USD 150K-160KApache Iceberg | Cloud Data | Cloud Data Warehouses | Data Lakes | Data Warehouses401k | Employee assistance program | Health spending account | Life insurance | Long-term disabilityMid-level ContractSunnyvale, California, USA R11h ago
-
Senior Embedded Engineer – Robotics & Sensor Systems USD 164K-220KAI Inference | AIS | AWS | Azure | C++Senior-level Full TimeArlington, VA11h ago
-
Mid-level Full TimeSan Francisco Bay Area11h ago
-
Senior Machine Learning (ML) Engineer USD 175K-200KAI Search | Artificial Intelligence | Attention Mechanisms | Autogen | Azure AI401k plan | Dental insurance | Flexible paid time off | Flexible work hours | Medical insuranceSenior-level Full TimeSeattle, WA R11h ago
-
Software Engineer, Data Runtime USD 29KAWS | Admission Policy | Arrow | Azure | Benchmarking401k | Additional company holidays | Enhanced parental leave | Generous vacation policy | Holiday breakSenior-level Full TimeSan Francisco, US (Hybrid) R11h ago
-
Applied Machine Learning Scientist (Remote) USD 128K-176KAlgorithms | Data Structures | Deep learning | Machine Learning | OptimizationConference attendance | Health benefits | Mental health care program | Paid time off | Parental leave programSenior-level Full TimeAlberta, Canada; British Columbia; Calgary, Alberta, … R12h ago
-
AI Engineer I USD 90K-119KAPI Integration | Azure | CI/CD | Docker | Embeddings401k match | Dental insurance | Holiday pay | Medical insurance | Paid time offEntry-level Full TimeRemote, CA, USA R12h ago
-
API | AWS | Automated testing | Cloud Computing | Code generation401k | Dental insurance | Discounts and perks | Fully remote | Medical insuranceSenior-level Full TimeUniversal City, CALIFORNIA, United States R12h ago
-
Staff Backend Engineer, Data Platform USD 140K-175KAsynchronous Service Design | Audit Logging | BigQuery | Cloud Run | Cloud Storage401k match | Company holidays | Dental insurance | Flexible paid time off | Health insuranceSenior-level Full TimeElizabeth, NJ R12h ago
-
Embedded Software Engineer - Drive Systems USD 150K-220KC# | C++ | CAN | ECU | FreeRTOSHealth insurance | Life insurance | Long-term care insurance | Long-term disability insurance | Paid time offMid-level Full TimeFoster City, CA13h ago
-
Senior Software Engineer (Data Platform) USD 185K-185K.NET | AWS | Authentication | Authorization Code | Authorization Code GrantSenior-level Full TimeUnited States (EST) R13h ago
-
Lead GenAI Engineer USD 150K-170KAgile | Azure | Cloud platform | Containerization | Cybersecurity401k | Employee assistance program | Health savings account | Life insurance | Long-term disabilitySenior-level ContractMinneapolis, Minnesota, USA R13h ago
-
API Development | Agent systems | BigQuery | Data integration | Databricks401k matching | Cell phone stipend | College Savings Assistance | Counseling services | Dental insuranceSenior-level Full TimeNew York, NY, United States14h ago
-
Data Scientist, Risk USD 186K-230KCausal Inference | Experimental Design | Python | Risk Modeling | SQLHybrid work optionsSenior-level Full TimeSan Francisco, CA - Hybrid R14h ago