Product Evaluations Lead - Gen AI Software
Tasks
- Build LLM evaluation datasets
- Create dashboards and executive readouts
- Define model release criteria
- Design metrics and perform failure analysis
- Develop evaluation frameworks and benchmarks
- Drive end to end model evaluation and analysis
- Partner with research and product teams
- Run partner experiments and A B tests
- Synthesize cross partner evaluation patterns into roadmap input
- Translate results into go no go recommendations
Perks/Benefits
- N/A
Skills/Tech-stack
A/B | A/B Testing | Agentic AI | B testing | Benchmarking | Causal Inference | Dataset Building | Experimentation | Human evaluation | LLM Evaluation | Language Models | Large Language Models | Metric Design | Model Evaluation | Multimodal AI | Python | Reward Models | Statistics
Education
Regions
Countries
States
Related jobs
-
C++ | CUDA | Data Preprocessing | Data Processing | Data loadingSenior-level Full TimeUS, CA, Santa Clara1d ago
-
C# | C++ | Caching | Distributed Systems | GPU ArchitectureComprehensive benefits | EquitySenior-level Full TimeUS, CA, Santa Clara R1d ago
-
Senior Research Scientist, Efficient Deep Learning USD 192K-304KAdaptive Inference | Architecture Search | C++ | CUDA | Computer VisionSenior-level Full TimeUS, CA, Santa Clara1d ago
-
Action Transformers | C++ | CUDA | Diffusion Models | Hardware PrototypingSenior-level Full TimeUS, CA, Santa Clara1d ago
-
Senior Deep Learning Frameworks Sustaining Engineer USD 152K-218KBazel | C# | C++ | CI Jenkins | CI/CDBenefits | EquitySenior-level Full TimeUS, CA, Santa Clara1d ago
-
Research Scientist, Human‑AI Perception and Interaction Research - PhD New College Grad 2026 USD 168K-264KAI Model Training | AI model | C++ | Deep learning | GPU TrainingEquity | Health benefits | Paid time offSenior-level Full TimeUS, CA, Santa Clara1d ago
-
C++ | CUDA | Deep learning | Generative AI | Machine LearningEquity | Health benefitsSenior-level Full TimeUS, CA, Santa Clara1d ago
-
Senior Research Scientist, Multi-Modal Language Models USD 192K-304KAlgorithms | Computer Vision | Data Structures | Data Synthesis | Distributed ComputingEquity | Health insurance | Paid time offSenior-level Full TimeUS, CA, Santa Clara1d ago
-
CUDA | Deep learning | Deep learning frameworks | Efficient Architectures | GPU ComputingBenefits | EquitySenior-level Full TimeUS, CA, Santa Clara1d ago
-
API | AWS | CI/CD | Cloud platform | Distributed SystemsSenior-level Full TimeSanta Clara, CA1d ago
-
Senior AI Workflow Engineer USD 184K-287KASP.NET | Agentic AI | Ansible | Artifactory | CI/CDEquity | Health insurance | Paid time offSenior-level Full TimeUS, CA, Santa Clara1d ago
-
AWS | Agent systems | Azure | C plus plus | Capacity PlanningSenior-level Full TimeUS, CA, Santa Clara1d ago
-
Senior Data Engineer, Engineering Data Analytics USD 168K-270KAWS Glue | Amazon Athena | Amazon EMR | Amazon Redshift | Amazon S3Equity | Health benefitsSenior-level Full TimeUS, CA, Santa Clara1d ago
-
Senior Software Engineer, CUDA Deep Learning Systems USD 184K-356KC++ | CUDA | Computer Architecture | Deep Learning Compilers | Deep learningBenefits | EquitySenior-level Full TimeUS, CA, Santa Clara R1d ago
-
Benchmarking | C++ | C++14 | CUDA | Compiler developmentSenior-level Full TimeUS, CA, Santa Clara1d ago
-
Algorithms | C++ | CUDA | Compilers | Computer ArchitectureSenior-level Full TimeUS, CA, Santa Clara1d ago
-
Azure DevOps | Bazel | C plus plus | C# | CI/CDSenior-level Full TimeUS, CA, Santa Clara1d ago
-
Software Engineer, CUDA Deep Learning Systems USD 124K-195KC++ | CUDA | Code generation | Computer Architecture | Deep Learning CompilersEquity | Health benefitsMid-level Full TimeUS, CA, Santa Clara R1d ago
-
Senior Deep Learning Algorithm Engineer USD 152K-241KAutoscaling | C++ | Deep learning | Distributed Systems | Inference OptimizationSenior-level Full TimeUS, CA, Santa Clara R1d ago
-
Senior Software Engineer, Machine Learning Inference USD 152K-241KC++ | CUDA | Compilers | Deep learning | GPU ProgrammingSenior-level Full TimeUS, CA, Santa Clara1d ago
-
Senior Machine Learning Engineer USD 184K-356KC++ | Computer Vision | Data pipeline | Deep learning | LidarSenior-level Full TimeUS, CA, Santa Clara1d ago
-
Senior Perception Engineer - Autonomous Vehicles USD 184K-356KC++ | Computer Vision | Data Pipelines | Deep learning | LidarBenefits | EquitySenior-level Full TimeUS, CA, Santa Clara1d ago
-
Senior Storage Production Engineer - DGX Cloud USD 184K-333KAccess Control | Anomaly Detection | Ansible | Auditing | BashSenior-level Full TimeUS, CA, Santa Clara R1d ago
-
Senior AI Systems and Algorithms Engineer USD 152K-241KDeep learning | Distributed Systems | Distributed Training | FSDP | GPU PerformanceSenior-level Full TimeUS, CA, Santa Clara R1d ago
-
C++ | CPU architecture | CUDA | Deep learning | Fraud DetectionSenior-level Full TimeUS, CA, Santa Clara R1d ago