Technical Program Manager III, AI/ML Data Analysis, Search
Tasks
- Apply classical test theory and item response theory
- Architect automated evaluation pipelines
- Communicate technical tradeoffs
- Conduct statistical calibration
- Design human evaluation systems
- Detect rater bias and data integrity threats
- Detect rater drift
- Establish inter rater reliability metrics
- Identify differential item functioning
- Identify risks
- Implement data quality checks
- Integrate human assessment workflows into model development lifecycle
- Lead complex multi-disciplinary projects
- Manage project schedules
- Plan requirements
- Translate latent constructs into behavioral assessments
Perks/Benefits
- N/A
Skills/Tech-stack
Bayesian Modeling | Classical Test Theory | Cohen Kappa | Computational Linguistics | Data Pipelines | Differential Item Functioning | Fleiss Kappa | Hierarchical modeling | Human Feedback | Inter-rater reliability | Intraclass Correlation | Item Response Theory | Language Models | Language Processing | Large Language Models | Learning from Human Feedback | Machine Learning | Multilevel modeling | Natural Language | Natural Language Processing | Program Management | Python | R | Reinforcement Learning | Reinforcement Learning from Human Feedback | Response Theory | Statistical Analysis | Statistical calibration | Test Theory
Education
Roles
Regions
Countries
States
Related jobs
-
Algorithm Design | Angular | Artificial Intelligence | Data Visualization | Full StackSenior-level Full TimeMountain View, CA, USA8h ago
-
Cross-Functional Collaboration | Cross-functional | Functional collaboration | Generative AI | OKRsSenior-level Full TimeBoulder, CO, USA; New York, NY, …8h ago
-
Code review | Computer Vision | Design Patterns | Language Models | Language ProcessingFlexible work arrangements | Inclusive team culture | Mentorship | Work-life balanceMid-level Full TimePasadena, California, USA19h ago
-
Head of Autonomous Driving Technology & AI Development USD 290K-400KAWS | Agile | Artificial Intelligence | Azure | C++401k match | Dental insurance | Flexible spending account | Health insurance | Health savings accountExecutive-level Full TimeSunnyvale, CA, United States21h ago
-
AI Solutions & Implementation Manager USD 180K-200KArtificial Intelligence | Budgeting | Business Process | Business Process Mapping | Business case401k | Commuter benefits | Dental insurance | Employee Lunch Perk | Flexible spending accountMid-level Full TimeNew York, NY22h ago
-
Senior Product Manager, Data Platform USD 170K-190KAI | Access Control | Business Intelligence | Compute Cost Optimization | Cost OptimizationDistributed team | Remote-first work environmentSenior-level Full TimeRemote - United States R23h ago
-
Data Analysis | Machine Learning | Optimization | Python | SQLAnnual bonus program | Company benefits program | Equity incentive plan | Hybrid work scheduleSenior-level Full TimeMountain View, CA, US; San Francisco, …23h ago
-
Manager, Data Strategy & Analytics USD 90K-115KA/B | A/B Testing | Adtech | Amazon Marketing Cloud | Amazon marketing401k contributions | Health insurance | Internet stipend | Learning stipend | Mobile phone stipendMid-level Full TimeNew York R1d ago
-
Engineering Manager, App SW - Embedded USD 252K-298KAdaptive AUTOSAR | Automated testing | Automotive SOC | Bash | Board SupportEquity package | Hybrid work policy | Paid time for office and home workMid-level Full TimeDetroit; Sunnyvale1d ago
-
Engineering Manager, App SW - Embedded USD 252K-298KAdaptive AUTOSAR | Automotive SOC | Bash | Board Support | Board Support PackageEquity compensation | Health insurance | Hybrid work | Paid time offMid-level Full TimeDetroit; Sunnyvale1d ago
-
Engineering Manager - Machine Learning USD 151K-203KAgent Orchestration | BigQuery | CUDA | Distributed Systems | DockerAnnual bonus | Comprehensive benefits package | Equity compensation | Hybrid work scheduleMid-level Full TimeSalt Lake City, Utah1d ago
-
Lead AI Operations Program Manager USD 163K-220KAI Agents | Artificial Intelligence | Automation | Bias Mitigation | Chain managementTravelSenior-level Full TimeSunnyvale, CA | Redmond, WA1d ago
-
Lead, Product Content Engineering USD 129K-187KAgile | Content Strategy | Data Annotation | Data Visualization | Dataset developmentSenior-level Full TimeMenlo Park, CA | New York, …1d ago
-
Global Product Lead, Global Business Applied AI USD 122K-175KAnalytics | Artificial Intelligence | Business Cases | Business Intelligence | CRMSenior-level Full TimeMountain View, CA, USA; San Francisco, …1d ago
-
Global Product Lead, Global Business Applied AI USD 149K-216KArtificial Intelligence | Data Analysis | Google Ads | Online Advertising | Product ManagementSenior-level Full TimeSan Francisco, CA, USA; Mountain View, …1d ago
-
Marketing Insights & Analytics Manager USD 105K-130KAttribution Modeling | Behavior Analysis | Channel measurement | Consumer Behavior Analysis | Consumer behaviorMid-level Full TimeBrookfield, WI, 53005, US1d ago
-
Product Manager, 3D Imagery & Analytics USD 120K-162K3D Reconstruction | API | Computer Vision | Data Visualization | Data pipeline401k matching | Commuter benefits | Dental insurance | Health insurance | High flexibilityMid-level Full TimeRedwood City, CA1d ago
-
Engineering Group Manager, AI/ML for Chassis Controls USD 217K-333KAgile | Artificial Intelligence | C plus plus | Chassis Controls | Control SystemsCompany vehicle | Hybrid work schedule | Relocation benefitsSenior-level Full TimeMountain View Technical Center - Mountain …1d ago
-
Product Manager, Agentic AI Solutions USD 138K-257KAPI | Access Management | Agentic Systems | Artificial Intelligence | Auditability401k match | Annual equity awards (eligibility) | Comprehensive benefits package | Hybrid work 3 days per week in office | Paid time offMid-level Full TimeCambridge (USA), United States1d ago
-
Audit Data Analyst - Audit Manager (Senior) USD 114K-218KAI Solutions | Alteryx | Archer | Automation | Continuous Reporting401k | Adoption Assistance | Career path planning | Continuing education | Dental insuranceSenior-level Full TimeCharlotte, NC - CENTS, United States1d ago
-
Executive-level Full TimeOne Destiny Way, Westlake TX, United …1d ago
-
AI Agents | AWS Neuron | Development Lifecycle | Generative AI | Machine LearningMid-level Full TimeNew York, New York, USA1d ago
-
Senior Director, Head of AI for Clinical Operations USD 203K-304KAgentic Systems | Benchmarking | Clinical trial | Clinical trial management | Clinical trial management systemsFlexible office schedule | In person collaboration days per weekSenior-level Full TimeUS - Boston - MA, United …1d ago
-
Group Engineering Manager, Algorithms & Software USD 85K-85K21 CFR Part 11 | 3D Reconstruction | AI/ML | C++ | CFR Part 11Mid-level Full TimeMassachusetts - Methuen, United States1d ago
-
Commercial Analytics Manager USD 149K-162KData Visualization | Data analytics | Power BI | Project Management | PythonCareer development | Dental insurance | Disability insurance | Flexible work models | Hybrid work optionMid-level Full TimeUS - California - Thousand Oaks, …1d ago