Senior Engineering Manager, Agentic & Generative AI Benchmarking and Evaluations
Santa Clara, California, United States
USD 201K-352K Senior-level Full Time
Tasks
- Benchmark frontier LLMs
- Build AI evaluation infrastructure
- Define enterprise AI benchmarks
- Optimize model selection for latency and cost
- Recruit and mentor AI evaluation team
- Translate evaluation results into product improvements and roadmap recommendations
- Validate grounding and RAG pipelines
Perks/Benefits
Skills/Tech-stack
CMDB | Context window | Context window optimization | Data analytics | Fine Tuning | Hybrid search | Integration Testing | Knowledge graphs | LLM Evaluation | Latency optimization | MLOps | Machine Learning | Model Fine-tuning | NumPy | Pandas | Prompt engineering | Python | Re-ranking | Retrieval-Augmented Generation | SQL | Semantic Re-ranking | Test automation | Unit Testing
Education
Regions
Countries
States
Cities
Related jobs
-
Senior Product Marketing Manager, AI Inference USD 180K-275KAI Model Benchmarking | AI model | Artificial Intelligence | Community Management | Editorial judgmentJob stability | Non-corporate work culture | Startup vitalitySenior-level Full TimeHeadquarters/Sunnyvale Office1d ago
-
Director / Lead, Commercial Analytics & Forecasting USD 195K-269KAccess Analytics | Claims analytics | Commercial Analytics | Coverage Analytics | DashboardingExecutive-level Full TimePrinceton, NJ, United States1d ago
-
Accounting Principles | Alteryx | Auditing Standards | Automation Anywhere | Continuous AuditingEntry-level Full TimeCoconut Grove, FL, United States1d ago
-
Business Analytics Manager, Revenue USD 125K-145KAI Agents | Business Intelligence | Data Modeling | Data Warehousing | Data analytics401k match | Dental insurance | Employee stock purchase plan | Flexible time off | Medical insuranceMid-level Full TimeHybrid - Denver R1d ago
-
Alteryx | Auditing Standards | Automation Anywhere | Business Intelligence | Continuous AuditingEntry-level Full TimeMiami, FL, United States1d ago
-
AI Computing | AI Systems Roadmap | AI infrastructure | AI systems | Cross-functionalSenior-level Full TimeSunnyvale, CA, USA2d ago
-
ClaimsXM Data Engineer III USD 125K-140KAnalysis Services | Azure | Azure Data | Azure Data Factory | Data Cleansing401k employer match | Health insurance | Paid Holidays | Paid Long Term Disability | Paid Short Term DisabilitySenior-level Full TimeUnited States2d ago
-
Senior Manager, Digital Analytics USD 95K-138KA/B | A/B Testing | Adobe Analytics | Anomaly Detection | Attribution401k company match | Health & wellness programs | Holiday schedule | Online and retail discounts | Paid time offSenior-level Full TimeFramingham, MA, United States2d ago
-
Project Manager – Data Science & AI Engineering USD 115K-142KAgile | Artificial Intelligence | Confluence | Data Science | JiraBusiness resource groups access | Learning platform access | Networking opportunities | One on one manager time | Professional development accessMid-level Full TimeSpringfield, Massachusetts; Boston, Massachusetts2d ago
-
Manager, Global Sourcing Analytics USD 75K-115KAI-enabled | AI-enabled tools | Amadeus | Analytics automation | Artificial Intelligence401k match | Employee Discounts on Cruises | Employee assistance program | Employee stock purchase plan | Health benefitsMid-level Full TimeMiami, FL, United States R2d ago
-
Analytics Lead, Safety Reporting USD 104K-131KDashboarding | Data Documentation | Data Quality | Data Reconciliation | Data analytics401k match | Child care benefits | Commuter benefits | Dental insurance | Family building benefitsSenior-level Full TimeSan Francisco, CA R2d ago
-
Manager, Data Analytics USD 160K-200KARR | BI Dashboards | Churn | Customer Engagement | Data GovernanceCompany swag | Long-term disability | Maternity leave | Medical/Dental/Vision insurance | Paternity leaveSenior-level Full TimeSalt Lake City, Utah2d ago
-
Analytics Manager USD 100K-125KBusiness Strategy | Consulting | Data Analysis | Data Modeling | Data VisualizationMid-level Full TimeJersey City, New Jersey, United States2d ago
-
Mid-level Full TimeNew York, NY, United States2d ago
-
AWS QuickSight | DBT | Data Analysis | Data Modeling | Data Quality401k benefits | Baby bonding leave | Commuter benefits | Dental insurance | Disability insuranceMid-level Full TimeUS Remote R2d ago
-
VP, Portfolio Analytics & Reporting USD 300K-405KAttribution | Capital reporting | Cash Flow | Cash Flow Reporting | Commitment Pacing401k match | Casual dress code | Childcare services | Community happy hours | Free catered foodExecutive-level Full TimeNew York, NY R2d ago
-
AWS | Ansible | Automation | BAM | Bioinformatics401k match | Commuter benefits | Employee referral program | Fertility care benefits | Free employee genetic testingSenior-level Full TimeUS Remote R2d ago
-
Technical Product Manager, AI Engineering & Systems USD 130K-170KAPI Design | Artificial Intelligence | Chatbot | Computer use | Data Analysis401k employer matching | Dental insurance | Health insurance | Paid parental leave | Paid time offMid-level Full TimeNew York, New York R2d ago
-
API Design | Agentic AI | Artificial Intelligence | Data Analysis | Experimentation401k matching | Dental insurance | Equity opportunities | Health insurance | Paid parental leaveSenior-level Full TimeNew York, New York R2d ago
-
Associate, Process Optimization USD 87K-114KData Analysis | Data metrics | Excel | Lean | Process Improvement401k plan | Life and disability insurance | Medical, dental, and vision insurance | Paid Holidays | Paid parental leaveMid-level Full TimeTempe, Arizona, United States R2d ago
-
Manager, Account Sales Analytics and Reporting USD 83K-133KDashboarding | Data Governance | Data Quality | Data Reconciliation | Data Validation401k plan | Childcare discounts | Commuter benefits | Dental insurance | Disability insuranceMid-level Full TimeBethesda, MD, United States R2d ago
-
Machine Learning Director USD 205K-301KAPI Design | Code Quality | Data Pipelines | ETL | Feature EngineeringWork-life balanceExecutive-level Full TimeSan Francisco, CA2d ago
-
Sr Manager, Data Engineering USD 135K-170KAgile | Amazon Web Services | Azure | Cloud Computing | Cost OptimizationSenior-level Full TimeClearwater, FL, United States2d ago
-
Associate Director of Analytics USD 110K-163KBusiness Intelligence | Cluster Analysis | Data Mining | Data Modeling | Data VisualizationMid-level Full TimeNashville, Tennessee, United States2d ago
-
Senior Manager, BizOps & Analytics USD 180K-230KClickHouse | Excel | GitHub | Jupyter | Metabase401k matching | Dental insurance | Health insurance | Paid parental leave | Paid time offSenior-level Full TimeNew York, New York R2d ago