Senior Manager, Data Quality & Evaluation

Tasks

• Build repeatable calibration and QA sampling processes
• Coach and develop quality operations team
• Conduct root cause analysis for quality variation
• Define evaluation frameworks and quality standards
• Design quality frameworks for AI data programs
• Develop rubrics and reviewer guidelines
• Identify and resolve quality risks affecting timelines
• Lead quality operations across human data workflows
• Lead quality operations for multilingual evaluation
• Manage quality acceptance criteria and review processes
• Manage quality reviews and customer reporting
• Monitor quality indicators and performance metrics
• Translate customer requirements into quality workflows
• Turn quality insights into process improvements

Perks/Benefits

• Fully remote
• International collaboration
• Professional growth

Skills/Tech-stack

Adjudication | Annotation quality | Audio evaluation | Calibration | Coding Evaluation | Data Quality | Error Analysis | Evaluation Design | Human Preference Evaluation | Human-in-the-loop | LLM Evaluation | Multilingual Evaluation | Performance reporting | Preference Evaluation | QA Sampling | Quality Assurance | Quality management | RLHF | Reviewer Training | Rubrics | Sampling strategy | Speech evaluation | The Loop | Transcription Evaluation