Technical Program Manager III, AI/ML Data Analysis, Search
Tasks
- Apply classical test theory and item response theory
- Architect automated evaluation pipelines
- Communicate technical tradeoffs
- Conduct statistical calibration
- Design human evaluation systems
- Detect rater bias and data integrity threats
- Detect rater drift
- Establish inter rater reliability metrics
- Identify differential item functioning
- Identify risks
- Implement data quality checks
- Integrate human assessment workflows into model development lifecycle
- Lead complex multi-disciplinary projects
- Manage project schedules
- Plan requirements
- Translate latent constructs into behavioral assessments
Perks/Benefits
- N/A
Skills/Tech-stack
Bayesian Modeling | Classical Test Theory | Cohen Kappa | Computational Linguistics | Data Pipelines | Differential Item Functioning | Fleiss Kappa | Hierarchical modeling | Human Feedback | Inter-rater reliability | Intraclass Correlation | Item Response Theory | Language Models | Language Processing | Large Language Models | Learning from Human Feedback | Machine Learning | Multilevel modeling | Natural Language | Natural Language Processing | Program Management | Python | R | Reinforcement Learning | Reinforcement Learning from Human Feedback | Response Theory | Statistical Analysis | Statistical calibration | Test Theory
Education
Roles
Regions
Countries
States
Related jobs
-
Product Manager, Machine Learning USD 171K-230KAd-Tech | Agent Orchestration | Analytics | Cross-Functional Collaboration | Cross-functionalMid-level Full TimeSeattle, WA9h ago
-
Algorithm Design | Angular | Artificial Intelligence | Data Visualization | Full StackSenior-level Full TimeMountain View, CA, USA9h ago
-
Cross-Functional Collaboration | Cross-functional | Functional collaboration | Generative AI | OKRsSenior-level Full TimeBoulder, CO, USA; New York, NY, …9h ago
-
Principal Technical Program Manager (Quantum Hardware) USD 165K-331KAI tools | Agile | Azure DevOps | Cryoelectronics | Data AnalysisSenior-level Full TimeRedmond, WA, US20h ago
-
Code review | Computer Vision | Design Patterns | Language Models | Language ProcessingFlexible work arrangements | Inclusive team culture | Mentorship | Work-life balanceMid-level Full TimePasadena, California, USA20h ago
-
Head of Autonomous Driving Technology & AI Development USD 290K-400KAWS | Agile | Artificial Intelligence | Azure | C++401k match | Dental insurance | Flexible spending account | Health insurance | Health savings accountExecutive-level Full TimeSunnyvale, CA, United States22h ago
-
AI Solutions & Implementation Manager USD 180K-200KArtificial Intelligence | Budgeting | Business Process | Business Process Mapping | Business case401k | Commuter benefits | Dental insurance | Employee Lunch Perk | Flexible spending accountMid-level Full TimeNew York, NY23h ago
-
Senior Product Manager, Data Platform USD 170K-190KAI | Access Control | Business Intelligence | Compute Cost Optimization | Cost OptimizationDistributed team | Remote-first work environmentSenior-level Full TimeRemote - United States R1d ago
-
Data Analysis | Machine Learning | Optimization | Python | SQLAnnual bonus program | Company benefits program | Equity incentive plan | Hybrid work scheduleSenior-level Full TimeMountain View, CA, US; San Francisco, …1d ago
-
Manager, Data Strategy & Analytics USD 90K-115KA/B | A/B Testing | Adtech | Amazon Marketing Cloud | Amazon marketing401k contributions | Health insurance | Internet stipend | Learning stipend | Mobile phone stipendMid-level Full TimeNew York R1d ago
-
Engineering Manager, App SW - Embedded USD 252K-298KAdaptive AUTOSAR | Automated testing | Automotive SOC | Bash | Board SupportEquity package | Hybrid work policy | Paid time for office and home workMid-level Full TimeDetroit; Sunnyvale1d ago
-
Engineering Manager, App SW - Embedded USD 252K-298KAdaptive AUTOSAR | Automotive SOC | Bash | Board Support | Board Support PackageEquity compensation | Health insurance | Hybrid work | Paid time offMid-level Full TimeDetroit; Sunnyvale1d ago
-
Machine Learning Engineering Manager, App SW USD 336K-381KAutonomous Driving | Autonomy systems | Controls engineering | Decision Making | Machine LearningCore working hours | Hybrid work | Work from homeMid-level Full TimeSunnyvale1d ago
-
Machine Learning Engineering Manager, App SW USD 336K-381KAutonomous Vehicles | Controls engineering | Decision Making | Machine Learning | PerceptionHybrid work policy | Inclusion and diversity culture | Work from homeMid-level Full TimeDetroit1d ago
-
Engineering Manager - Machine Learning USD 151K-203KAgent Orchestration | BigQuery | CUDA | Distributed Systems | DockerAnnual bonus | Comprehensive benefits package | Equity compensation | Hybrid work scheduleMid-level Full TimeSalt Lake City, Utah1d ago
-
P&C Insurance Analytics Lead USD 150K-200KApache Spark | Artificial Intelligence | Azure | Data Ingestion | Data ModelingClient site travelSenior-level Full TimeUnited States1d ago
-
Curriculum Manager - Cloud USD 110K-142KAWS | Azure | Cloud Architecture | Cloud Computing | Cloud DataMid-level Full TimeBelgium; Portugal; United Kingdom; United States1d ago
-
Curriculum Manager - Cloud USD 110K-142KAWS | Agentic Coding systems | Agentic coding | Azure | Claude CodeMid-level Full TimeBelgium; Portugal; United Kingdom; United States1d ago
-
Curriculum Manager - Cloud USD 110K-142KAWS | Agentic Coding systems | Agentic coding | Azure | Cloud ArchitectureMid-level Full TimeBelgium; Portugal; United Kingdom; United States1d ago
-
Curriculum Manager - Cloud USD 110K-142KAWS | Azure | Cloud Architecture | Cloud Data | Cloud Data EngineeringMid-level Full TimeBelgium; Portugal; United Kingdom; United States1d ago
-
Lead AI Operations Program Manager USD 163K-220KAI Agents | Artificial Intelligence | Automation | Bias Mitigation | Chain managementTravelSenior-level Full TimeSunnyvale, CA | Redmond, WA1d ago
-
Lead, Product Content Engineering USD 129K-187KAgile | Content Strategy | Data Annotation | Data Visualization | Dataset developmentSenior-level Full TimeMenlo Park, CA | New York, …1d ago
-
Global Product Lead, Global Business Applied AI USD 122K-175KAnalytics | Artificial Intelligence | Business Cases | Business Intelligence | CRMSenior-level Full TimeMountain View, CA, USA; San Francisco, …1d ago
-
Global Product Lead, Global Business Applied AI USD 149K-216KArtificial Intelligence | Data Analysis | Google Ads | Online Advertising | Product ManagementSenior-level Full TimeSan Francisco, CA, USA; Mountain View, …1d ago
-
Marketing Insights & Analytics Manager USD 105K-130KAttribution Modeling | Behavior Analysis | Channel measurement | Consumer Behavior Analysis | Consumer behaviorMid-level Full TimeBrookfield, WI, 53005, US1d ago