Senior Engineering Manager, Agentic & Generative AI Benchmarking and Evaluations
Santa Clara, California, United States
USD 201K-352K Senior-level Full Time
Tasks
- Benchmark frontier LLMs
- Build AI evaluation infrastructure
- Define enterprise AI benchmarks
- Optimize model selection for latency and cost
- Recruit and mentor AI evaluation team
- Translate evaluation results into product improvements and roadmap recommendations
- Validate grounding and RAG pipelines
Perks/Benefits
Skills/Tech-stack
CMDB | Context window | Context window optimization | Data analytics | Fine Tuning | Hybrid search | Integration Testing | Knowledge graphs | LLM Evaluation | Latency optimization | MLOps | Machine Learning | Model Fine-tuning | NumPy | Pandas | Prompt engineering | Python | Re-ranking | Retrieval-Augmented Generation | SQL | Semantic Re-ranking | Test automation | Unit Testing
Education
Regions
Countries
States
Cities
Related jobs
-
Featured Feat. Associate Director, Data Labs USD 167K-167KAWS | Cloud Computing | Compute Infrastructure | Data Analysis | LLM GovernanceConference speaking opportunities | Hybrid work schedule | Media appearancesSenior-level Full TimeWashington, District of Columbia, 20004, United … R17d ago
-
Manager, Sourcing Analytics & Strategy USD 110K-120KBudgeting | Data Analysis | Forecasting | Google Sheets | JavaScript401k | Customer Discount | Dental insurance | Disability insurance | Flexible spending accountSenior-level Full TimeChicago, IL11h ago
-
Senior Manager, Product Management - Text Analytics USD 253K-333KClassification | Cost Latency Accuracy Modeling | Fine Tuning | Language Models | Language ProcessingDedicated growth time | Team-building activities | Tech stipend | Wellness and mental health benefitsSenior-level Full TimeSeattle, Washington, United States14h ago
-
Data Analytics Manager, Elections USD 130K-185KDashboards | Data Quality | Data Visualization | ETL | Generative AI401k | Dental insurance | Medical insurance | Paid leave | Tuition reimbursementMid-level Full TimeNew York, NEW YORK, United States16h ago
-
Commercial Analytics Senior Manager USD 130K-165KA/B | A/B Testing | Attribution | B testing | CI/CDFormal and informal training opportunities | Health and dental benefits | Income protection | Life insurance | Long-term disabilitySenior-level Full TimeBrookfield, CT, 06804, USA23h ago
-
Senior Advanced Analytics & Insights Manager USD 107K-199KAWS | Azure | Classification | Cloud Computing | Clustering401k match | Dental insurance | Medical insurance | Unlimited paid time off | Vision insuranceSenior-level Full TimeFramingham, MA1d ago
-
AI | Build and release | Communication | Cross-Functional Collaboration | Cross-functionalMid-level Full TimeCupertino1d ago
-
Sr Mgr - AI Engagement & Transformation USD 163K-204KADMS | AI Governance | Business Analysis | Business Process | Business process improvement401k match | Career advancement | Dental insurance | Health insurance | Performance bonusesSenior-level Full TimeCT - Orange - 180 Marsh …1d ago
-
Actuarial Science | Data analytics | Insurance | Loss reserving | MS Office401k matching | Employee discounts | Health insurance | Mental health counseling | Paid HolidaysSenior-level Full TimeHartford - Tower, United States1d ago
-
Manager, Data Analytics & Reporting USD 87K-157KAudit management | Business Objects | CMS reporting | Data Analysis | Data Mining401k | Flexible work arrangement | Health insurance | Hybrid work options | Paid time offMid-level Full TimeRemote-FL, United States R1d ago
-
Manager of Data Engineering USD 52K-52KAWS Glue | Apache Airflow | Apache Kafka | Buildkite | CI/CD401k matching | Company holidays | Medical coverage | Paid time off | Paid volunteer daysSenior-level Full TimeOH Columbus 7400 Safelite Way, United …1d ago
-
Analytics | Artificial Intelligence | Cloud Computing | Data Governance | Data ProductsGrowth opportunities | Mentorship | Onsite collaboration days per week | Training and mentorshipSenior-level Full TimeUSA-MN Mounds View South, United States1d ago
-
Quality Applications Manager - Quality Data Analytics USD 120K-180KAudit Support | Automation | Business Objects | Data Quality | Data VisualizationMid-level Full TimeUSA-MN Mounds View South, United States1d ago
-
Data Scientist / Manager USD 113K-164KDashboarding | Data Cleansing | Data Visualization | Data analytics | Machine LearningDisability insurance | Health insurance | Life insurance | Paid leave | Professional developmentMid-level Full TimeAsheville, NC1d ago
-
Mid-level Full TimeChicago, IL2d ago
-
Data Scientist Associate Manager - Remote USD 134K-209KAI Foundry | AWS Bedrock | Agentic AI | Azure AI | Azure AI FoundryConnectivity reimbursement | Technology equipment provided | Work from homeMid-level Full TimeUSA - IL (Remote), United States R2d ago
-
Operations Manager, AI/ML Data Solutions USD 136K-197KArtificial Intelligence | Compliance | Data Analysis | Data Visualization | Data labelingSenior-level Full TimeSan Bruno, CA, USA; New York, …2d ago
-
Automation | Backend Data Manipulation | Backend data | BigQuery | BusinessObjectsMid-level Full TimeSunnyvale, CA, USA2d ago
-
AI Governance | Agile | Analytics Platforms | Data Management | Data analyticsBackup childcare | Financial coaching | Health care coverage | Mental health support | On-site health and wellness centersExecutive-level Full TimeNY, United States3d ago
-
Lead, Data Quality - Partnerships USD 100K-160KAudit | Calibration | Contract Management | Data Quality | Data labelingSenior-level Full TimeSan Jose, CA3d ago
-
Head of Data USD 193K-230KAPI Integration | BigQuery | CDC | Cloud Storage | DBTEquity compensation | Quarterly visits to center | Relocation benefits | Remote workExecutive-level Full TimeLos Angeles or San Francisco; Remote … R3d ago
-
Senior Engagement Manager, AI Solutions Delivery USD 116K-207KArtificial Intelligence | Bias monitoring | Data Quality | Data integration | Ethical AIFlexible work schedule | Travel opportunitiesSenior-level Full TimeUSA.NY.Home Office NY Metro Tri-State, United …3d ago
-
Senior AI Program Manager USD 140K-180KAI | AI Automations | API Integration | Agentic Workflows | Claude401k | Casual dress code | Commuter benefits | Dental coverage | Flexible vacationSenior-level Full TimeSeattle, WA3d ago
-
Apache Flink | Apache Iceberg | Apache Kafka | Apache Spark | Audio ProcessingSenior-level Full TimeSeattle3d ago
-
Manager, Portfolio Analytics Architect USD 114K-142KAI Automation | Databricks | ETL | LLM | PythonSenior-level Full TimeSalt Lake City, UT3d ago