Senior Manager, Data Quality & Evaluation
Tasks
- Conduct quality reviews and support customer escalations and retrospectives
- Define evaluation frameworks and quality standards
- Design AI data quality frameworks
- Develop rubrics task instructions and reviewer guidelines
- Implement quality improvements across tooling training staffing and workflows
- Lead quality operations for multilingual evaluation speech audio review transcription and annotation
- Manage quality processes for calibration QA sampling adjudication and reporting
- Monitor quality indicators and error trends
- Partner with internal teams for quality success from planning to production
- Perform root cause analysis for quality variation
- Translate customer requirements into quality workflows
Perks/Benefits
- Fully remote
- International team collaboration
- Opportunity to Lead AI Evaluation Operations
- Professional growth opportunities
Skills/Tech-stack
Adjudication | Annotation Guidelines | Audio Review | Calibration | Cause analysis | Coding Evaluation | Data Quality | Error Analysis | Evaluation rubrics | Human Preference Evaluation | Human review | Human review workflows | Human-in-the-loop | LLM Evaluation | Multilingual Evaluation | Performance Metrics | Preference Evaluation | QA Sampling | Quality Assurance | RLHF | Reporting | Review workflows | Reviewer Training | Root Cause Analysis | Root cause | Scorecards | Speech evaluation | The Loop | Transcription
Education
N/A
Related jobs
-
Automation | Big Data | CI/CD | Cloud infrastructure | Cloud platformFlexibility to work from anywhere | High-impact leadership | International team collaboration | Remote-first | Work-life balanceExecutive-level Full TimeGermany R6d ago
-
Senior Product Manager - Analytics USD 150K-200KAI | Analytics | ClickHouse | Columnar Databases | Data VisualizationSenior-level Full TimeGermany (Remote) ; Ireland (Remote); Netherlands … R11d ago
-
BI & Strategic Analytics Lead (m/f/d) EUR 60K-72KAWS | Agile | Amazon Redshift | Azure | Azure SynapseEquity compensation | Hybrid remote work | Professional development supportSenior-level Full TimeGermany - Remote R15d ago
-
AI Data Engineering Lead, Vice President GBP 90K-126KAgent Orchestration | Agent systems | Agile | Artificial Intelligence | ConfluenceSenior-level Full TimeL14 - Verde Building, 10 Bressenden … R22d ago
-
Senior Product Manager (AI Analytics) GBP 91K-120KAPIs | Analytics | Data Analysis | Experimentation | MetricsExtra time off | Flexible work location | Internal events | Language classes | Learning and development opportunitiesSenior-level Full TimeAmsterdam, Netherlands; Belgrade, Serbia; Berlin, Germany; … R24d ago
-
Automation | CRM | Conversion Rate | Conversion rate optimization | DashboardingFlexible work | Paid unlimited vacation | Remote-first | Team events | Transparent cultureSenior-level Full TimeDeutschland - Remote R1mo ago