AIML - Sr Manager, Evaluation - Data Science & Insights
Tasks
- Build and deploy LLM autograders
- Collaborate with cross-functional teams
- Communicate evaluation results to leaders
- Create rubrics for model evaluation
- Develop evaluation frameworks for AI features
- Drive proactive improvements in models and agentic features
- Lead data science evaluation team
- Manage AI quality measurement instrumentation
Perks/Benefits
- N/A
Skills/Tech-stack
Data Quality | Data Quality Measurement | Data Science | Evaluation Frameworks | Human evaluation | Instrumentation | LLM Autograders | Language Models | Large Language Models | Logging | Machine Learning | Privacy | Quality Measurement | Rubrics | Statistical Analysis
Education
Related jobs
-
Machine Learning Engineer Graduate (E-Commerce Supply Chain & Logistics - LLM/Agent) - 2027 Start (PhD) USD 137K-246KAgents | Benchmarking | C++ | Deep learning | EvaluationEntry-level Full TimeSan Jose, California, United States1h ago
-
Machine Learning Engineer Graduate (E-Commerce Supply Chain & Logistics - LLM/Agent) - 2027 Start (PhD) USD 137K-246KAI Feedback | Agents | Benchmarking | C++ | DPOEntry-level Full TimeSeattle, Washington, United States1h ago
-
Agent evaluation | Classification | Clustering | Data Mining | Deep learningEntry-level Full TimeSeattle, Washington, United States1h ago
-
Machine Learning Engineer Intern (E-Commerce Supply Chain & Logistics-LLM / Agent) - 2027 Start (PhD) USD 137K-246KBenchmarking | C++ | Context understanding | Direct Preference Optimization | Evidence GroundingEntry-level InternshipSeattle, Washington, United States1h ago
-
Machine Learning Engineer Intern (E-Commerce Supply Chain & Logistics-LLM/Agent) - 2027 Start (PhD) USD 137K-246KBenchmarking | C++ | DPO | Deep learning | EvaluationEntry-level InternshipSeattle, Washington, United States1h ago
-
Classification | Clustering | Data Mining | Deep learning | E-commerceEntry-level Full TimeSan Jose, California, United States1h ago
-
Machine Learning Engineer (Technical Leadership) USD 213K-251KAB Testing | C plus plus | Data Mining | Data Pipelines | Deep learningSenior-level Full TimeMenlo Park, CA | New York, …2h ago
-
Software Developer, Scaled Ops AI Acceleration Team USD 170K-209KAgent Orchestration | Artificial Intelligence | Backend Development | Bug tracking | ComplianceSenior-level Full TimeAustin, TX | Menlo Park, CA …2h ago
-
Research Engineer, Machine Perception (Project Aria) USD 141K-208K3D Computer Vision | C++ | Computer Vision | Data Pipelines | Deep learningOnsite work | Work authorization supportMid-level Full TimeRedmond, WA2h ago
-
Cross-Functional Collaboration | Cross-functional | Dashboarding | Functional collaboration | Machine LearningMid-level Full TimeNew York, NY, USA; Reston, VA, …2h ago
-
Software Engineer, Managed Service for Apache Spark USD 147K-210KApache Hadoop | Apache Spark | Apache YARN | BigQuery | Cloud platformMid-level Full TimeKirkland, WA, USA2h ago
-
Algorithms | Android | Android AppCompat | C# | C++Mid-level Full TimeWaterloo, ON, Canada; San Jose, CA, …2h ago
-
3D Perception | C++ | Camera Calibration | Computer Vision | Data StructuresSenior-level Full TimeSeattle, WA, USA; Mountain View, CA, …2h ago
-
Staff Software Engineer, Applied AI, Model Quality USD 207K-300KData Processing | Data Storage | Data Structures | Data Structures and Algorithms | DebuggingSenior-level Full TimeNew York, NY, USA2h ago
-
AI | Analytics | Computer Vision | Cross-Functional Collaboration | Cross-functionalSenior-level Full TimeReston, VA, USA; Atlanta, GA, USA2h ago
-
Software Engineer III, AI/ML GenAI, Search USD 147K-210KC++ | Data Processing | Debugging | Information Retrieval | LLMSenior-level Full TimeNew York, NY, USA2h ago
-
C++ | Cloud Computing | Data Processing | Debugging | Distributed ComputingSenior-level Full TimeMountain View, CA, USA2h ago
-
Senior Staff Software Engineer, AI/ML, Google Cloud USD 262K-364KAccelerated computing | Artificial Intelligence | Compute Engine | Data Processing | Data StorageSenior-level Full TimeSeattle, WA, USA; Sunnyvale, CA, USA2h ago
-
Photonic Engineer, Machine Learning USD 188K-274KCircuit simulation | Data center | Data center networking | Digital Signal | Digital Signal ProcessingSenior-level Full TimeSunnyvale, CA, USA2h ago
-
AI Inference | Benchmarking | C++ | Collective communication | Distributed SystemsSenior-level Full TimeMountain View, CA, USA2h ago
-
Software Engineer III, Generative AI, Core USD 147K-210KCode optimization | Compiler design | Computer Science | Concurrent programming | Generative AISenior-level Full TimeKirkland, WA, USA2h ago
-
Engineering Lead, Google Labs USD 262K-364KAI Foundation | AI foundation models | C plus plus | C# | Distributed ComputingBonus | Equity | Health insurance | Paid time off | Retirement benefitsSenior-level Full TimeMountain View, CA, USA; San Francisco, …2h ago
-
Sr Data Engineer_ITDE1I USD 102K-170KAPI Integration | AWS | Automation | Azure | CI/CD401k match | Dental insurance | Disability insurance | Hybrid work policy | Life insuranceSenior-level Full TimeHOUSTON, US, 77056 R5h ago
-
Sr. Automation Engineer Robotics Software USD 115K-144KAutomation | Computer Vision | Databases | FastAPI | HMIDental insurance | Health insurance | Paid time off | Retirement plan | Sick leaveSenior-level Full TimeAnkeny, Iowa, US R5h ago
-
Machine Learning Engineer, Memory USD 125K-194KAgile | Computer Vision | Foundation Models | Horizon Modeling | Language Processing401k match | Coaching | Dental insurance | Life insurance | Long-term disabilityMid-level Full TimeCary,North Carolina,United States8h ago