Senior Manager, Data Quality & Evaluation

Tasks

• Build calibration QA sampling adjudication and reviewer performance tracking processes
• Convert quality insights into operational improvements and training
• Define evaluation standards acceptance criteria review processes metrics
• Design quality frameworks for AI data and evaluation programs
• Develop rubrics task instructions reviewer guidelines and scorecards
• Identify quality risks and resolve issues with delivery teams
• Lead coach and develop quality managers and reviewers
• Lead multilingual speech audio transcription and coding evaluation quality operations
• Manage data collection annotation evaluation and review processes
• Monitor quality indicators reviewer agreement error trends and root causes
• Support customer programs with quality reviews escalations retrospectives and improvement plans
• Translate customer requirements into quality workflows

Perks/Benefits

• Flexibility to work from anywhere
• Fully remote
• International team collaboration
• Professional growth opportunities

Skills/Tech-stack

Acceptance criteria | Adjudication | Annotation | Annotation Guidelines | Audio evaluation | Calibration | Coding Evaluation | Data Quality | Data collection | Error Analysis | Evaluation Frameworks | Golden datasets | Human-in-the-loop | LLM Evaluation | Multilingual Evaluation | Performance reporting | QA Sampling | Quality Assurance | Quality Standards | RLHF | Reviewer Training | Rubrics | Scorecards | Speech evaluation | The Loop | Transcription | Trust and Safety