Senior Engineering Manager, Agentic & Generative AI Benchmarking and Evaluations
Santa Clara, California, United States
USD 201K-352K Senior-level Full Time
Tasks
- Benchmark frontier LLMs
- Build AI evaluation infrastructure
- Define enterprise AI benchmarks
- Optimize model selection for latency and cost
- Recruit and mentor AI evaluation team
- Translate evaluation results into product improvements and roadmap recommendations
- Validate grounding and RAG pipelines
Perks/Benefits
Skills/Tech-stack
CMDB | Context window | Context window optimization | Data analytics | Fine Tuning | Hybrid search | Integration Testing | Knowledge graphs | LLM Evaluation | Latency optimization | MLOps | Machine Learning | Model Fine-tuning | NumPy | Pandas | Prompt engineering | Python | Re-ranking | Retrieval-Augmented Generation | SQL | Semantic Re-ranking | Test automation | Unit Testing
Education
Regions
Countries
States
Cities
Related jobs
-
AI Technology and Strategic Planning Researcher USD 155K-175KAI Governance | AI ethics | Artificial Intelligence | Autonomous Driving | Embodied intelligenceSenior-level Full Time圣何塞4h ago
-
Enterprise Analytics Lead USD 119K-222KABAP Managed CDS | Access Control | Analysis Services | Analytics Cloud | Azure Synapse9/80 work schedule | Extra day off every other week | In office schedule in Portsmouth NHSenior-level Full TimePortsmouth, NH, United States6h ago
-
Artificial Intelligence | Business Process | Business process improvement | CRM | Digital TransformationMid-level Full TimeParsippany, NJ, United States7h ago
-
US - Program Manager - Data Solutions (AI PMO USD 130K-180KADFDI | Azure DevOps | Data Cleansing | Data Governance | Data ManagementSenior-level Full TimeIrvine, United States8h ago
-
Acceptance Testing | Access Control | Access Management | Active Directory | Business Intelligence401k match | Dental insurance | Financial education resources | Life insurance | Long-term disabilityMid-level Full TimeCT-ROCKY HILL, 500 ENTERPRISE DR, 3RD …22h ago
-
Manager, Workday Reporting & Analytics USD 121K-205KAI | Adaptive Planning | Advanced Reports | BIRT | Calculated FieldsSenior-level Full TimeChicago (North Wacker Drive), United States22h ago
-
Business Partner and Product Management Manager (NG&NGL Digital Operations & Optimization ) USD 141K-212KAgile | Artificial Intelligence | Cloud Computing | Cybersecurity | Data GovernanceMid-level Full TimeFindlay OH Main Bldg, United States22h ago
-
Cash Flow | Discounted cash flow | Git | Interest Rate | Interest Rate Term Structure401k match | Adoption Assistance | Disability insurance | Life insurance | Medical, dental, and vision insuranceMid-level Full TimeCharlotte, NC, United States22h ago
-
AI Inference | AI orchestration | Agent systems | Agentic AI | DeepSeekDental insurance | Employee assistance program | Flexible paid vacation and sick leave | Flexible spending account | Health insuranceSenior-level Full TimeFTC03 - Ft. Collins, CO B-3 …22h ago
-
Senior Manager, Data Engineering USD 107K-150KAWS | Agile | Apache Airflow | Cloud Data | Cloud Data MigrationHybrid work environment | Occasional travel for stakeholder engagementSenior-level Full TimeMcKinney, TX, United States22h ago
-
Authentication Flows | Call Steering | Conversational AI | Customer Experience | Customer Experience Optimization401k plan | Adoption reimbursement | Disability benefits | Employee Assistance Programs EAP | Employee discountsSenior-level Full TimeAtlanta, Georgia22h ago
-
Analytics frameworks | Authentication Flows | Caller Intent Capture | Contact center | Contact center analytics401k plan | Adoption reimbursement | Disability benefits | Employee assistance programs | Employee discountsSenior-level Full TimeDallas, Texas22h ago
-
Senior Manager - Data Analytics USD 106K-203KAzure Data | Azure Data Lake | Azure Synapse | Dashboards | Data AnalysisSenior-level Full TimeUnited States, Washington, Redmond1d ago
-
Director of Engineering, AI/ML USD 137K-386KAWS SageMaker | Cloud deployment | Deep learning | Fine Tuning | Generative AI401k match | Company recognition program | Dental insurance | Education assistance | Flexible work locationExecutive-level Full TimeRemote, United States R1d ago
-
Staff Product Manager, AI/ML USD 195K-210KArtificial Intelligence | Data analytics | Enterprise Software | Machine Learning | MetricsSenior-level Full TimeNew York1d ago
-
Software Engineering Manager - Perception Occupancy USD 196K-280KAutonomous Systems | Computer Vision | Computer Vision Systems | Diffusion Models | Geofence ExpansionSenior-level Full TimeFoster City, CA1d ago
-
AI Workflow Orchestration | AI workflow | APIs | Data Structures | Evaluation and monitoring401k matching | Childcare reimbursement | Commuter reimbursement | Dental insurance covered | Medical Insurance CoveredEntry-level Full TimeNew York Office1d ago
-
Head of Engineering USD 200K-280KAWS | Data Processing | Distributed Systems | FFmpeg | Machine Learning401k access | Flexible PTO | Health, dental, vision coverage | Paid mealsExecutive-level Full TimeSan Francisco, CA, US2d ago
-
Product Manager II, YouTube GenAI Responsibility USD 163K-236KA/B | A/B Testing | AI Safety | B testing | Content ModerationEquity compensation | Health benefits | Paid time off | Retirement planMid-level Full TimeSan Bruno, CA, USA2d ago
-
Engineering Manager, Data Residency and Fast Compliance USD 207K-300KAutomation | Batch Processing | Cloud Computing | Cloud platform | ComplianceSenior-level Full TimeSunnyvale, CA, USA2d ago
-
Artificial Intelligence | Data Pipelines | Data analytics | Machine Learning | SAPMid-level Full TimeAtlanta, GA, USA2d ago
-
Agent systems | Artificial Intelligence | Cloud platform | Context engineering | Data GovernanceSenior-level Full TimeSan Francisco, CA, USA; Sunnyvale, CA, …2d ago
-
Engineering Director, Spatial Flex USD 307K-427KAI machine learning | Capacity Management | Capacity Planning | Cloud infrastructure | Data CompressionExecutive-level Full TimeSunnyvale, CA, USA; Kirkland, WA, USA2d ago
-
Engagement Manager, AI Solutions - Federal USD 150K-250KArtificial Intelligence | Data Science | Federal Contracting | Machine Learning | Program ManagementMid-level Full TimeUnited States (Remote) R2d ago
-
Manager, Software Engineering, Machine Learning USD 170K-277KCaching | Data Modeling | Deep learning | Fine Tuning | GPUHealth and wellness programs | Time away from workMid-level Full TimeMountain View, CA, United States2d ago