AI Quality and Agent Operations Analyst - AI Solutions
Tasks
- Analyze evaluation results service metrics production data user feedback
- Assess risks from AI outputs and recommend controls human review
- Define test scenarios and evaluation criteria
- Develop evaluation datasets and quality reports
- Establish quality metrics and acceptance thresholds
- Evaluate AI outputs for accuracy relevance safety
- Investigate failures root cause and performance trends
- Replace ad hoc testing with standard AI evaluation
- Support production readiness and release testing validation monitoring
Perks/Benefits
- N/A
Skills/Tech-stack
APIs | Case design | Cause analysis | Data Analysis | Data workflows | Generative AI | Incident Management | Information Systems | Machine Learning | Observability | Prompt evaluation | Python | Quality Assurance | Retrieval-Augmented Generation | Root Cause Analysis | Root cause | SQL | Service Metrics | Test Case | Test Case Design | Test Harness
Education
Bachelor of Engineering | Bachelor of Science | Bachelor of Science in Computer Science | Bachelor of Science in Data Science | Bachelor of Science in Information Systems
Roles
AI | AI Operations Analyst | AI Quality Analyst | Analyst | Operations Analyst | Quality Analyst
Regions
Countries
States
Related jobs
-
Applied AI Innovation Engineer — Sales AI Applications USD 124K-195KAI Governance | APIs | Agentic Workflows | Artificial Intelligence | BenchmarkingEmployee benefits | EquityMid-level Full TimeUS, CA, Santa Clara1d ago
-
Lead AI Data Engineer – Platform & Analytics BRL 40K-40KAWS Athena | Apache Airflow | Apache Iceberg | DAGs | DBT CoreSenior-level Full TimeUS,CA,Santa Clara2d ago
-
Lead ASIC Package Design Automation and AI Engineer USD 160K-264K2.5D IC | 3D-IC | Artificial Intelligence | Auto-routing | C++Senior-level Full TimeUS,CA,Santa Clara3d ago
-
2027 Masters Data Analyst Intern/co-op USD 69K-105KAKS | Azure | Data Engineering | Data Modeling | Data ProcessingEntry-level InternshipUS,CA,Santa Clara R7d ago
-
2027 Undergrad Data Analyst Intern/co-op USD 69K-106KAKS | Azure | Data Modeling | Data Visualization | Database DesignEntry-level InternshipUS,CA,Santa Clara R7d ago
-
Senior AI Infrastructure Software Engineer - DGX Cloud USD 184K-287KC# | C++ | Cause analysis | Debugging | Distributed SystemsBenefits | EquitySenior-level Full TimeUS, CA, Santa Clara R8d ago
-
Benchmarking | Calibration | Data Pipelines | FP4 | INT8Computational resources | Dinners | Fun activities | Lunches | SnacksSenior-level Full TimeSanta Clara, CA14d ago
-
Senior Applied Machine Learning Engineer - VLSI Design USD 152K-230KASIC | Agentic Systems | C++ | Circuit optimization | Data AnalysisSenior-level Full TimeUS, CA, Santa Clara16d ago
-
API | AWS | CI/CD | Cloud platform | Distributed SystemsSenior-level Full TimeSanta Clara, CA23d ago
-
Engineering Manager, AI Developer Technology USD 184K-356KC plus plus | CPU architecture | CUDA | Computer Architecture | Deep learningEmployee benefits | Equity | Hybrid workSenior-level Full TimeUS, CA, Santa Clara23d ago
-
Sr Staff IT AI Software Engineer USD 145K-235KA2A | Agent SDK | Cloud platform | Containerization | DjangoSenior-level Full TimeSanta Clara, CA25d ago
-
Senior Solutions Architect, Generative AI USD 184K-287KCollective communication | Congestion Control | Containers | DCGM | Data PipelinesConferencing Tools Access | Occasional travel | Remote workSenior-level Full TimeUS, CA, Santa Clara R29d ago
-
AI Researcher - Efficient AI (Contractor) USD 90K-114KAgent systems | Cache Compression | DPO | Distillation | Hybrid Attention401k matching | Fitness Goal Incentives | Health, dental, vision insurance | Hybrid work | Life and disability insuranceEntry-level ContractSanta Clara, CA R1mo ago
-
Lead Generative AI Analyst - English US USD 122K-186KData Analysis | Data labeling | Documentation | Machine Learning | Process ImprovementSenior-level Full TimeSanta Clara, CA1mo ago
-
AI/ML Engineer - Library Optimization USD 164K-264KArtificial Intelligence | Benchmarking | Characterization | Circuit design | Digital circuitSenior-level Full TimeSanta Clara1mo ago