Senior Technical Program Manager, Human-in-the-Loop Operations
Tasks
- Define and track SLAs and KPIs
- Deliver datasets to model teams
- Drive continuous process improvement
- Implement AI assisted pre labeling workflows
- Implement LLM as a judge evaluation
- Lead data programs end to end
- Maintain labeling guidelines and annotation schemas
- Manage distributed contractor workforce allocation
- Measure throughput inter annotator agreement cost per label
- Oversee contractor execution and quality review
- Own labeling and model evaluation workflows
- Partner with ML engineers and data scientists
- Route automated QA for labeling
- Translate ML requirements into annotation tasks
Perks/Benefits
Skills/Tech-stack
AI-assisted workflows | Annotation Schema | Automated QA | Data Operations | Data Quality | Data Quality Metrics | Data labeling | Inter-annotator agreement | KPI Tracking | LLM | LLM Pre Labeling | LLM-as-a-Judge | Machine Learning | Python | Quality Control | Quality metrics | SLAs | SQL
Education
N/A
Regions
Countries
States
Cities
Related jobs
-
Practice Architect II AI/ML USD 148K-222KAI | Change Control | Contract Management | Costing | Cross-functional401k | Employee assistance program | Health savings account | Life insurance | Long-term disabilitySenior-level Full TimeSunnyvale, California, USA R1d ago
-
Senior Manager, Data Science - Styling Algorithms USD 200K-246KA/B | A/B Testing | Agentic AI | Artificial Intelligence | B testingAnnual bonus | Dental insurance | Equity grants | Health insurance | Restricted stock unitsSenior-level Full TimeRemote USA R1d ago
-
Director, Enterprise Analytics USD 171K-185KConfound Control | Experimentation | Incrementality Measurement | Performance Marketing | Power analysisHealth benefits | Hybrid work scheduleExecutive-level Full TimeCambridge, Massachusetts, United States R2d ago
-
Senior-level Full TimeMiami, United States R2d ago
-
Analytics Manager GBP 55K-65KDashboard Development | Data Governance | Data Quality | Data Visualization | KPIFlexible benefits | Healthcare benefits | On-target bonus | Paid Holidays | Pension planMid-level Full TimeAny of our offices R2d ago
-
Senior Analytics Manager GBP 63K-80KAI | Analytics | Business Intelligence | Data Governance | Data MaturityCritical illness cover | Dental insurance | Discounts | Extra holiday buying | Flexibility to work from homeSenior-level Full TimeAny of our offices R2d ago
-
Manager, Data Engineering USD 200K-335KApache Airflow | Apache NiFi | Apache Spark | BigQuery | Cloud platformSenior-level Full TimeUS - San Francisco R2d ago
-
PMO Sr Mgr Data Gov Analytics USD 122K-212KAgile | Data Catalog | Data Engineering | Data Governance | Data fabricSenior-level Full TimeUS159 NJ Titusville - 1125 Trenton-Harbourton … R2d ago
-
Senior Engineering Manager, Embedded Software USD 136K-252KAI | Automation | Board Bring-up | Bring-up | C#401k thrift plan | Disability programs | Flexible working hours | Life insurance | Private medical careSenior-level Full TimeUS-NY-SKANEATELES-721 VISIONS DRIVE, United States R2d ago
-
Electronic Health Records | Health Records | Mixed methods | R | Research methodsFamily-friendly workplace | Remote/hybrid work arrangement | Training mentoring professional development opportunitiesMid-level Full TimePressey Hall (0309), United States R2d ago
-
Quality Manager, Data & Insights Analytics USD 60K-159KCRM | Dashboarding | Data analytics | Interaction Analytics | Medicare AdvantageDental coverage | Medical coverage | Paid time off | Remote work support | Retirement savings optionsMid-level Full TimeWork At Home-Texas, United States R2d ago
-
Statistical Process Control (SPC) Manager USD 109K-159KAdvanced Analytics | Capability analysis | Change Management | Control Charts | Data VisualizationCareer development opportunities | Employee resource groups | Fitness reimbursement | Flexible work schedules | Matching 401kMid-level Full Time222 Chicago IL, United States R2d ago
-
Head of AI & Program Technology Engineering USD 185K-200KAgentic Systems | Artificial Intelligence | Azure | Backup and Recovery | CI/CDSenior-level Full TimeNew York, NY HQ USA, United … R2d ago
-
Manager, Data Analysis - PULSE Risk & Compliance USD 149K-171KAWS | Cause analysis | Compliance | Data Architecture | Data GovernanceHealth benefits | Hybrid work | Inclusive workplace | Performance-Based Incentives | Sponsorship for work authorizationSenior-level Full TimeHouston, TX, United States R2d ago
-
Engineering Manager, Data Transformation USD 214K-321KAirflow | Amazon S3 | Flink SQL | GRPC | HiveMid-level Full TimeUS-SEA, US-SF, US-NYC, US-Remote R2d ago
-
Director of Analytics, Merchant Ecosystem USD 180K-220KA/B | A/B Testing | Accuracy evaluation | Agentic Analytics | B testing401k matching | Cell phone stipend | EAP | FSA | HSAExecutive-level Full TimeLos Angeles, CA, US R2d ago
-
Manager, Machine Learning Engineering USD 219K-329KAPIs | AWS | Apache Spark | CI/CD | Code review401k matching | Dental benefits | Employee recognition programs | Flexible time off | Healthcare benefitsSenior-level Full TimeSan Francisco, CA R2d ago
-
Associate Director of Statistical Programming USD 100K-146KADaM | AI integration | Bash | CAPA | CDISCContinuing education | Hybrid work environment | Professional meeting attendance | Work-life balanceMid-level Full TimeBoston, MA, United States R2d ago
-
Engineering Manager, Data Platform USD 290K-330KAWS | Apache Airflow | Apache Spark | DBT | Data Ingestion401k | Commuter benefits | Employee assistance program | Free Lunches | Medical/Dental/Vision insuranceMid-level Full TimeSan Francisco, CA R2d ago
-
Senior Quantum Chemistry Product Manager USD 176K-207KArtificial Intelligence | Computational Chemistry | Go-to-market | Machine Learning | Product Requirements401k match | Disability insurance | Equity grants | Health insurance | Life insuranceSenior-level Full TimePalo Alto, California, United States R2d ago
-
Technical Program Manager USD 140K-170KAI Agents | API Integration | Data Architecture | Data Governance | Data Leakage PreventionMid-level Full TimeRemote, United States R2d ago
-
Technical Product Manager USD 80K-100KA/B Testing | B testing | Cost Optimization | Experiment design | ExperimentationMid-level Full TimeUnited States R2d ago
-
Senior Manager of Data Engineer USD 162K-180KAWS | DBT | Data Governance | Data Quality | Data Security12-week paid parental leave | 4 week paid sabbatical | 401k match | Flexible PTO | Health benefitsSenior-level Full TimeRemote, US R3d ago
-
Senior Manager, SEM Analytics USD 119K-150KAccount structure | Automated bidding | Bid strategy | Budget Pacing | Budget allocationHealth and wellbeing benefits | Hybrid work scheduleSenior-level Full TimeCambridge, Massachusetts, United States R3d ago
-
Manager, Data Analytics USD 111K-145KCommon Table Expression | Dashboard Development | Data Modeling | Data Visualization | Excel11 paid holidays | 401k plan | Annual performance bonuses | Life and disability insurance | Medical, dental, and vision benefitsMid-level Full TimeNew York, New York, United States R3d ago