Product Evaluations Lead - Gen AI Software
Tasks
- Conduct partner experimentation and A B testing
- Create evaluation datasets and human evaluation strategies
- Define measurable model release criteria
- Design metrics and perform failure analysis
- Develop evaluation frameworks and benchmarks
- Drive end to end model evaluation and analysis
- Report findings via dashboards and executive readouts
- Translate model behavior into go no go recommendations
Perks/Benefits
Skills/Tech-stack
A/B | A/B Testing | B testing | Benchmarking | Causal Inference | Failure analysis | Generative AI | LLM Evaluation | Language Models | Large Language Models | Machine Learning | Metric Design | Multimodal AI | Python | Reward Models | Statistics
Education
Roles
Data Scientist | Evaluation Lead | Lead | Product Evaluation Lead | Scientist
Regions
Countries
States
Related jobs
-
C++ | Computer Vision | Deep learning | Distributed Training | Human robotEmployee benefits | EquitySenior-level Full TimeUS, CA, Santa Clara4d ago
-
Senior Applied AI Researcher, Digital Biology USD 184K-356KAgent systems | Agentic AI | Artificial Intelligence | C++ | CUDAEquity | Health benefitsSenior-level Full TimeUS, CA, Santa Clara14d ago
-
ARM Cortex | ARM Cortex-M | BMC | Bash | C plus plusSenior-level Full TimeUS, CA, Santa Clara20d ago
-
Technical Lead, GenAI - Autonomous Vehicles USD 224K-356KAWS | Azure | C++ | Cloud Computing | ControlSenior-level Full TimeUS, CA, Santa Clara20d ago
-
Principal Data Scientist - Cloud Gaming and AI USD 248K-379KAgent systems | Anomaly Detection | CI/CD | Cause analysis | Constraint OptimizationSenior-level Full TimeUS, CA, Santa Clara R21d ago