Senior Manager, Data Quality & Evaluation
Tasks
- Build quality workflows for data collection annotation and review
- Define evaluation standards and acceptance criteria
- Design quality frameworks for AI data and evaluation
- Improve rubrics task instructions and reviewer guidelines
- Lead coach and develop quality operations teams
- Lead multilingual speech and audio evaluation and transcription quality
- Manage calibration sampling and QA adjudication
- Manage reviewer performance tracking and customer reporting
- Monitor quality indicators and performance metrics
- Perform error analysis and root cause identification
- Turn quality insights into process training and tooling improvements
Perks/Benefits
- Fully remote
- International team collaboration
- Professional growth opportunities
- Remote work flexibility
Skills/Tech-stack
Adjudication | Annotation Guidelines | Audio evaluation | Calibration | Data Quality | Error Analysis | Evaluation Design | Human Preferences | Human-in-the-loop | LLM Evaluation | Multilingual Evaluation | Performance Metrics | QA methodologies | Quality Assurance | RLHF | Reviewer Training | Rubrics | Sampling strategies | Speech evaluation | The Loop | Transcription Evaluation
Education
N/A
Related jobs
-
Privacy & Analytics Manager EUR 30K-40KAdobe Analytics | CMP configuration | Cloud platform | Consent Mode | Cookie-less trackingFlexible compensation plan | Flexible working hours | Hybrid work model | Meal subsidy | Office coffee and fruitMid-level Full TimeMadrid, Spain R2d ago
-
Benchmarking | Context engineering | Cybersecurity | Deep learning | Deterministic systemsCareer growth | Equity package | Flexible work across Europe | Fully remoteExecutive-level Full TimeSpain R3d ago
-
Automation | Big Data | CI/CD | Cloud platform | Data ArchitectureFlexibility to work from anywhere | International team collaboration | Remote-first work environment | Work-life balance cultureExecutive-level Full TimeSpain R6d ago
-
Data Governance Consultant EUR 35K-40KAgile | Change Management | Confluence | Data Glossary | Data GovernanceAgile environment | Career development | Indefinite contract | Intensive work schedule in July and August | Telework supportMid-level Full TimeBarcelona, Spain R16d ago
-
AI Engineer EUR 50K-70KAWS Bedrock | AWS EventBridge | AWS Lambda | AWS SageMaker | AWS Step FunctionsContinuous learning plans | Hybrid work model | Lunch and coffee provided | Paid time off | Shortened FridaysMid-level Full TimeES - Alicante, Spain R28d ago
-
Staff/Senior AI Engineer, AI for Code GBP 81K-115KAgentic Workflows | Benchmarking | Context engineering | Cost Optimization | Fine TuningExtra time off | Flexible work location | Internal events | Language classes | Learning and development opportunitiesSenior-level Full TimeAmsterdam, Netherlands; Belgrade, Serbia; Berlin, Germany; … R28d ago
-
A/B | A/B Testing | AWS | Anthropic API | AzureFlexible location | Remote work | Stock optionsSenior-level Full TimeBarcelona office R1mo ago