Senior Manager, Data Quality & Evaluation

Tasks

• Build calibration QA sampling and adjudication processes
• Conduct customer quality reviews and escalations
• Define evaluation frameworks and standards
• Design quality frameworks for AI data
• Develop rubrics and reviewer guidelines
• Identify and resolve quality risks
• Lead coach and develop quality operations teams
• Lead multilingual speech audio transcription and annotation quality operations
• Manage data collection annotation evaluation review calibration reporting
• Monitor quality indicators and root causes
• Translate customer requirements into quality workflows
• Turn quality insights into process improvements

Perks/Benefits

• Fully remote
• International team collaboration
• Professional growth opportunities

Skills/Tech-stack

Adjudication | Annotation Guidelines | Annotation platforms | Audio evaluation | Calibration | Cause analysis | Dashboards | Data Annotation | Data workflows | Error Analysis | Evaluation rubrics | Human-in-the-loop | LLM Evaluation | Localization Quality | Multilingual Evaluation | Performance reporting | QA Methodology | QA Tools | Quality Assurance | Quality management | RLHF | Reviewer Training | Root Cause Analysis | Root cause | Sampling strategies | Scorecards | Speech Audio | Speech Audio Evaluation | The Loop | Transcription