Senior Manager, Data Quality & Evaluation

Tasks

• Build QA sampling and adjudication processes
• Calibrate reviewers and track reviewer performance
• Coach and develop quality managers and reviewers
• Define evaluation standards and acceptance criteria
• Design quality frameworks for AI data
• Develop rubrics and reviewer guidelines
• Drive quality improvements to tooling training and staffing models
• Lead quality operations for annotation and evaluation programs
• Monitor quality indicators and error trends
• Perform root cause analysis for quality variation
• Run multilingual evaluation and speech audio review operations
• Support customer quality reviews and escalations
• Translate customer requirements into quality workflows

Perks/Benefits

• Fully remote
• International team collaboration
• Leadership influence on quality strategy
• Professional growth opportunities

Skills/Tech-stack

Adjudication | Annotation Guidelines | Audio evaluation | Calibration | Cause analysis | Coding Evaluation | Data Annotation | Error Analysis | Evaluation | Human-in-the-loop | LLM Evaluation | Multilingual Evaluation | Performance reporting | Quality Assurance | Quality management | RLHF | Reviewer Training | Root Cause Analysis | Root cause | Rubrics | Sampling strategy | Speech Audio | Speech Audio Evaluation | The Loop | Transcription