Senior Technical Program Manager, Human-in-the-Loop Operations
Tasks
- Define and track SLAs and KPIs
- Deliver datasets to model teams
- Drive continuous process improvement
- Implement AI assisted pre labeling workflows
- Implement LLM as a judge evaluation
- Lead data programs end to end
- Maintain labeling guidelines and annotation schemas
- Manage distributed contractor workforce allocation
- Measure throughput inter annotator agreement cost per label
- Oversee contractor execution and quality review
- Own labeling and model evaluation workflows
- Partner with ML engineers and data scientists
- Route automated QA for labeling
- Translate ML requirements into annotation tasks
Perks/Benefits
Skills/Tech-stack
AI-assisted workflows | Annotation Schema | Automated QA | Data Operations | Data Quality | Data Quality Metrics | Data labeling | Inter-annotator agreement | KPI Tracking | LLM | LLM Pre Labeling | LLM-as-a-Judge | Machine Learning | Python | Quality Control | Quality metrics | SLAs | SQL
Education
N/A
Regions
Countries
States
Cities
Related jobs
-
Applied AI Architect, Enterprise Tech USD 240K-315KAPI Integration | Cloud Architecture | Data Science | Enterprise Integration | LLM EvaluationFlexible working hours | Generous vacation | Hybrid work policy | Optional equity donation matching | Parental leaveSenior-level Full TimeSan Francisco, CA | New York … R20h ago
-
Staff Data Architect USD 157K-192KBatch Processing | Cloud modernization | Cloud platform | Data Architecture | Data Contracts401k match | Dental insurance | Financial wellness | Health insurance | Hybrid workSenior-level Full TimeChicago, IL 60607, United States R22h ago
-
Head of Applied Machine Learning USD 292K-343KData Science | Embeddings | Evaluation metrics | Experimentation | Feature EngineeringInclusive work environment | Professional growth and mentorshipExecutive-level Full TimeUnited States - Remote R1d ago
-
Senior Data Analyst - Product USD 95K-115KAI | Business Analysis | Data Warehousing | Project Management | SDLC401k plan | Employee wellbeing committee | Generous time off | Health benefits | Hybrid work modelSenior-level Full TimeNew York, NY | Austin, TX R1d ago
-
Senior Machine Learning Engineering Manager USD 120K-190KAWS | Algorithms | Azure | Data Exploration | Design PatternsSenior-level Full TimeCharlotte, North Carolina, United States; Denver, … R1d ago
-
Embedded Software V&V Manager USD 176K-316KAUTOSAR | Agile | Automated testing | Azure DevOps | C#Employee resource groups | Flexible family care days | Option to purchase additional vacation time | Paid Holidays | Paid parental leaveSenior-level Full TimePalo Alto, CA, United States R1d ago
-
Product Manager - Data & Analytics USD 136K-190KAcceptance criteria | Agile | Amazon Redshift | Business Intelligence | Data Lake401k matching | Dental insurance | Employee referral program | FSA | Health insuranceSenior-level Full TimeOverland Park, KS R1d ago
-
Senior Manager, Data Engineering USD 104K-123KAgile | Alerting | Azure DevOps | CI/CD | Cause analysisSenior-level Full TimeN/A, N/A, United States R1d ago
-
Connected Data Engineer USD 86K-144K5-why | 8D | Analytics Dashboards | Automated detection | Big DataAdoption & Surrogacy Reimbursement | Community service time off | Employee resource groups | Fertility treatment benefits | Flexible family care daysMid-level Full TimeDearborn, MI, United States R1d ago
-
APIs | CRM | Data Monitoring | GraphQL | HubSpot workflows401k with company matching | Childcare reimbursement | Commuter reimbursement | Generous parental leave policy | Medical, dental, and vision insuranceMid-level Full TimeSan Francisco Office R1d ago
-
Director, Data Quality & Adoption USD 106K-116KAcceptance Testing | Change Management | Coaching | Continuous Improvement | Customer SuccessComprehensive benefits package | Domestic travel | Paid time off | Virtual work environmentExecutive-level Full TimeRemote, VA, US R1d ago
-
Senior, Engineering Manager - Perception & Auto-Tagging USD 195K-234K2D Object Detection | 3D Object Detection | AWS | Auto-labeling | Cloud infrastructure401k employer match | Accidental death insurance | Dental insurance | Flexible schedule | Health insuranceSenior-level Full TimeAnn Arbor, MI, Remote - US R1d ago
-
Big Data | Customer Success | Data Architecture | Data Engineering | Data IntelligenceRemote work optionsSenior-level Full TimeIllinois; Massachusetts; New York; Washington, D.C. R1d ago
-
Digital Analytics Manager Sr USD 113K-186KA/B | A/B Testing | Adobe Analytics | Adobe Experience | Adobe Experience PlatformMid-level Full TimeWork From Home United States, United … R1d ago
-
Digital Analytics Manager USD 89K-146KA/B | A/B Testing | Adobe Analytics | Adobe Launch | B testingMid-level Full TimeWork From Home United States, United … R1d ago
-
Analytics Manager - Strategic Analytics USD 92K-135KBI reporting | Commercial Analytics | Customer Analytics | Data Modeling | Data VisualizationHybrid/remote optionsMid-level Full TimeUS: Norwalk, United States R1d ago
-
Manager, Enterprise Data Engineer USD 125K-225KADLS | AI Search | Azure AI | Azure AI Search | Azure DataTelecommuting | Travel 25 percentMid-level Full TimeManhattan, United States R1d ago
-
API Design | Agent integration | Cross-Functional Collaboration | Cross-functional | Data InfrastructureSenior-level Full TimeSan Jose, United States R1d ago
-
Legal AI Program Manager USD 315K-385KArtificial Intelligence | Change Management | Enterprise Architecture | Enterprise Software | EvaluationMid-level Full TimeRemote, United States; UK - Remote R1d ago
-
Communications Insights & Analytics Mgr USD 95K-153KAnalytics | Artificial Intelligence | Automation | BigQuery | Business IntelligenceSenior-level Full TimeHouston, United States R1d ago
-
Senior Manager, Pricing Strategy & Analytics USD 203K-251KBilling | Business Intelligence | CPQ | CRM | Cause analysisAdoption leave | Commuter benefits | Dental insurance | Disability insurance | Flexible spendingSenior-level Full TimeSan Mateo, CA, United States R2d ago
-
Manager II, Data Platform USD 154K-234KAWS | AWS DMS | AWS Fargate | Amazon DynamoDB | Amazon ECSComprehensive health insurance | Flexible remote work | Parental leave | Professional development stipendMid-level Full TimeRemote - US R2d ago
-
Manager - Data Engineering USD 75K-100KAgile methodology | Airflow | Automated testing | BigQuery | Campaign Performance401(k) retirement benefits | Casual dress code | Company events | Employee discount | Health benefitsMid-level Full TimePittsburgh, PA, United States R2d ago
-
Associate Director, Analytics USD 110K-140KBayesian Methods | Bias analysis | Causal analysis | Confidence Intervals | Data Quality401k matching | Extended parental leave | Flexible PTO | Fully covered health insurance | Half-day FridaysMid-level Full TimeNew York City Metropolitan Area R2d ago
-
AI Data | AI data strategy | Airflow | Alerting | Analytics engineeringEmployee benefits | Hybrid work flexibilitySenior-level Full TimeBoston, MA R2d ago