Staff Machine Learning Engineer, Agent Eval Platform
Mountain View, CALIFORNIA, United States
R
USD 176K-308K (estimate) Senior-level Full Time
Tasks
- Build agent evaluation platform
- Build reward model for process reward
- Create per item rubrics and validation
- Design and calibrate judge models
- Develop production-grade Python code
- Enable offline to online divergence analysis
- Fine tune judge models
- Generate calibrated confidence scores
- Implement simulated world and versioned scenarios
- Optimize agent using reward model
- Route uncertain judgments to humans
Perks/Benefits
Skills/Tech-stack
AI Feedback | Calibration | Experimentation | Fine Tuning | Human Feedback | Human annotation | Information Retrieval | LLM Evaluation | Language Models | Large Language Models | Learning from Human Feedback | Machine Learning | Offline/Online Evaluation | Online evaluation | Prompt engineering | Python | Ranking | Recommender Systems | Reinforcement Learning | Reinforcement Learning from AI Feedback | Reinforcement Learning from Human Feedback | Reranking | Reward Modeling
Education
N/A
Regions
Countries
States
Related jobs
-
C++ | Deep learning | Distributed Training | ETL | GoSenior-level Full TimeMountain View, CALIFORNIA, United States R5h ago
-
Embedded Software Engineer III USD 100K-135KC# | C++ | Containerization | Driver Development | Inspection tools401k retirement plan | Bereavement leave | Dental insurance | FMLA | Medical insuranceSenior-level Full TimeCentennial, CO, United States R9h ago
-
Embedded Software Engineer II USD 90K-115KC# | C++ | Linux | MAC addresses | Memory Management401k retirement plan | Bereavement leave | Dental insurance | FMLA | Health insuranceMid-level Full TimeCentennial, CO, United States R9h ago
-
Ai Engineer USD 160K-170KAgentic Workflows | Anomaly Detection | Context engineering | Experimentation | Language Models401k | Employee assistance program | Health Spending Account HSA | Life insurance | Long-term disabilityMid-level ContractPalo Alto, California, USA R11h ago
-
Software Engineer Level 1-FFPP-8933 USD 78K-250KAWS Glue | AWS Lambda | Amazon DynamoDB | Amazon Web Services | BSON401k match | Bonus opportunities | Dental insurance | Employee referral program | Flexible spending accountsSenior-level Full TimeHanover, MD; Information Technology R12h ago
-
Software Engineer Level 2-FFPP-8934 USD 78K-250KAWS | Accumulo | BSON | HBase | HDFS401k match | Dental insurance | Employee referral programs | Flexible spending accounts | Flexible work arrangementsSenior-level Full TimeHanover, MD; Information Technology R12h ago
-
Data Engineer, Splunk & Snowflake USD 150K-160KAnomaly Detection | Apache Iceberg | Apache Spark | Business Intelligence | Cloud Data401k retirement plan | Employee assistance program | Health spending account | Life insurance | Long-term disabilityMid-level ContractSunnyvale, California, USA R12h ago
-
AI Systems & ML Engineering Industry Expert USD 140K-209KA/B | A/B Testing | API Design | AWS | Agentic SystemsFlexible schedule | Fully remote | Hourly payment | International teamSenior-level Part TimeNew York City, NY, US R14h ago
-
Machine Learning Engineer USD 100K-120KAWS | Apache Spark | Classification | Data Quality | Data Quality ChecksHybrid work model | Onsite 1 to 2 days per week | Opportunities for growthMid-level Full TimeOakbrook Terrace, IL, US R16h ago
-
Senior Machine Learning Engineer, Payments USD 191K-223KAirflow | Anomaly Detection | Data Processing | Deep learning | Fine TuningSenior-level Full TimeRemote-USA R1d ago
-
Senior Data Engineer (Data Architecture) USD 105K-135KAPIs | AWS Glue | Airflow | Azure Data | Azure Data Factory401k matching | Dental insurance | Extended holiday break | Flexible schedules | Life insuranceSenior-level Full TimeRemote, United States R1d ago
-
Data Engineer, Hardware Reliability (Starlink) USD 135K-185KBash | C# | C++ | Concurrency | Data InfrastructureExtended hours and weekends | Onsite work required | Travel up to 10%Mid-level Full TimeBastrop, TX R1d ago
-
Apache Airflow | Apache Flink | Apache Iceberg | Apache Kafka | Apache Spark401k employer match | Comprehensive healthcare benefits | Family planning support | Flexible vacation | Gender-affirming careSenior-level Full TimeRemote - United States R1d ago
-
Forward Deployed Engineer (Generative AI) USD 180K-230KAI Agent Builder | AI Pipelines | Agent Builder | Agent Developer Kit | Agentic designCareer development | Equal employment opportunities | Individual responsibilitySenior-level Full TimeAustin, Texas, United States - Remote R1d ago
-
Performance & Data Optimization Engineer USD 128K-173KAWS | ArgoCD | Automated failover | Automation | Azure BicepMid-level Full TimeUSA LA Home Office (LAHOME), United … R1d ago
-
Senior Software Engineer, AI/ML (Hybrid - Aguadilla) USD 107K-204KAWS | Airflow | Apache Iceberg | CI/CD | Dagster401k match | Dental insurance | Disability coverage | Employee assistance program | Hybrid work scheduleSenior-level Full TimeUS-PR-AGUADILLA-110 ~ Rd 110 N Km … R1d ago
-
AI Software Engineer USD 76K-76KAPI Integration | APIs | Apex | CI/CD | CRMRemote work flexibilityMid-level Full TimeAppliance Park KY US, United States R1d ago
-
Cost Optimization | Distributed Training | Execution orchestration | FSDP | GPU efficiencySenior-level Full TimeSan Jose, United States R1d ago
-
AWS | Apache Arrow | Apache Ray | Apache Spark | AzureSenior-level Full TimeSan Jose, United States R1d ago
-
Quantitative Trading Engineer (Java) - Vice President USD 155K-252KCode review | Continuous Delivery | E-trading | Java | KDBEmployee resource groups | Family building benefits | Generous vacation | Health and wellbeing benefits | Hybrid work modelExecutive-level Full TimeNew York, 1 Columbus Circle, United … R1d ago
-
Sr. Scientific Data Engineer, R&D Data Platform USD 78K-156KAWS | AWS Glue | AWS Lambda | Amazon Athena | Amazon EMRSenior-level Full TimeUnited States of America : Remote R1d ago
-
Staff Software Engineer, Data Platform USD 231K-340KAWS | Access Control | Airbyte | Airflow | Anomaly DetectionSenior-level Full TimeNew York R1d ago
-
Staff Software Engineer, Data Platform USD 231K-340KAWS | Access Control | Airbyte | Amazon Redshift | Apache AirflowSenior-level Full TimeSan Francisco R1d ago
-
Sr. Staff Software Engineer, Big Data Platform USD 177K-364KAWS | Apache Flink | Apache Presto | Apache Spark | Data ProcessingIn office collaboration 1 to 2 times per quarterSenior-level Full TimePalo Alto; Seattle, WA; New York, … R1d ago
-
Senior AI Engineer, VideoAI USD 144K-236KA/B | A/B Testing | Apache Spark | Artificial Intelligence | B testingHealth and wellness programs | Time offSenior-level Full TimeMountain View, CA, United States R1d ago