Technical Program Manager III, AI/ML Data Analysis, Search
Tasks
- Apply classical test theory and item response theory
- Architect automated evaluation pipelines
- Communicate technical tradeoffs
- Conduct statistical calibration
- Design human evaluation systems
- Detect rater bias and data integrity threats
- Detect rater drift
- Establish inter rater reliability metrics
- Identify differential item functioning
- Identify risks
- Implement data quality checks
- Integrate human assessment workflows into model development lifecycle
- Lead complex multi-disciplinary projects
- Manage project schedules
- Plan requirements
- Translate latent constructs into behavioral assessments
Perks/Benefits
- N/A
Skills/Tech-stack
Bayesian Modeling | Classical Test Theory | Cohen Kappa | Computational Linguistics | Data Pipelines | Differential Item Functioning | Fleiss Kappa | Hierarchical modeling | Human Feedback | Inter-rater reliability | Intraclass Correlation | Item Response Theory | Language Models | Language Processing | Large Language Models | Learning from Human Feedback | Machine Learning | Multilevel modeling | Natural Language | Natural Language Processing | Program Management | Python | R | Reinforcement Learning | Reinforcement Learning from Human Feedback | Response Theory | Statistical Analysis | Statistical calibration | Test Theory
Education
Roles
Regions
Countries
States
Related jobs
-
Senior Manager, Data Platform & Autonomy Infrastructure USD 225K-275KAutonomy | Data Engineering | Data Modeling | Data Pipelines | Data StorageDental insurance | Equity compensation | Health insurance | Paid time off | Performance bonusSenior-level Full TimeSouth San Francisco, California, USA8h ago
-
Staff Product Manager, Agentic AI Applications USD 172K-258KAPI Design | Agentic Systems | Artificial Intelligence | CI/CD | Command LineSenior-level Full TimeMountain View, California; San Francisco, California11h ago
-
Machine Learning Engineering Manager, Critical Harms USD 295K-345KAutomation | Data Analysis | Distributed Systems | Engineering Management | ExperimentationEquity compensation | Health benefits | Onsite work schedule guidanceMid-level Full TimeSan Mateo, CA, United States R13h ago
-
Manager, Analytics - Business Insights USD 100K-151KAgile | Automation | BigQuery | DBT | Data ModelingMid-level Full TimeCharlotte, North Carolina, United States; Denver, …13h ago
-
Data Manager - Senior (KP) Job#818 USD 129K-155KAI machine learning | Agile | Cloud Architecture | Data Pipelines | Data VisualizationSenior-level Full TimeSpringfield, Virginia, United States16h ago
-
Staff Product Manager, AI/ML USD 250K-270KA/B | A/B Testing | B testing | Data Pipelines | ExperimentationHybrid work modelSenior-level Full TimeStrava SF R19h ago
-
Senior-level Full TimeCenter, Center District, IL19h ago
-
Manager, Forward Deployed Engineering - CMEG USD 211K-291KApache Spark | Data Governance | Data Migration | Databricks | Delta LakeMid-level Full TimeRemote - Washington D.C. R19h ago
-
Manager, Forward Deployed Engineering - Manufacturing USD 211K-291KAnalytics | Apache Spark | Data Migration | Databricks | Delta LakeMid-level Full TimeRemote - California R19h ago
-
ASICs | Agent Orchestration | Artificial Intelligence | Bias Mitigation | CommunicationMid-level Full TimeSunnyvale, CA | Bellevue, WA | …22h ago
-
DRAM | Design for test | Design verification | HBM | Machine LearningSenior-level Full TimeSunnyvale, CA, USA22h ago
-
Social Insights and Analytics Manager, AI Marketing USD 141K-206KAI | Audience engagement | Dashboarding | Data Analysis | ForecastingSenior-level Full TimeSan Francisco, CA, USA; New York, …22h ago
-
Product Manager - Data & Analytics USD 110K-145KAgile | Amazon Web Services | Cloud Computing | Cloud platform | Data Architecture401k | Dental insurance | Discounts | Medical insurance | Paid leaveMid-level Full TimeUniversal City, CALIFORNIA, United States R1d ago
-
Principal Program Manager, Commercial AI Programs USD 105K-190KAI Platform | AI Platform Program Management | Agile | Budget Management | Commercial Go to Market401k | Employee stock grants | Employee stock purchase plan | Flexible spending account | Home internet discountsSenior-level Full TimeWA-Headquarters, Building 3, United States1d ago
-
Manager, Cybersecurity - Process and Analytics USD 126K-228KAI Driven | AI-driven detection | Artificial Intelligence | Automation workflows | Cybersecurity401k | Commuter and transit programs | Employee stock grants | Employee stock purchase plan | Home internet discountMid-level Full TimeKS-Headquarters, Building 6360 OP, United States1d ago
-
People Analytics Manager - Workday USD 100K-150KAdaptive Planning | Composite Reports | Correlation | Data Governance | Data QualityAccident insurance | Critical illness insurance | Dental insurance | Disability benefits | Hybrid work arrangementMid-level Full TimeRemote - Illinois, United States R1d ago
-
Data Science Manager USD 140K-160KAgile | Artificial Intelligence | CPLEX | Data Mining | Data VisualizationMid-level Full TimeAtlanta, GA1d ago
-
AlertMedia | Corporate Resilience | Crisis management | Emergency response | EverbridgeSenior-level Full TimeCalifornia - San Francisco, United States1d ago
-
Manager, Fraud Detection and Analytics USD 83K-158KAnomaly Detection | Card Fraud | Data Analysis | Data Visualization | Fraud DetectionMid-level Full TimeOne Destiny Way, Westlake TX, United …1d ago
-
Quant Analytics Manager - Credit Risk Modeling USD 116K-216KCloud Computing | Data Modeling | Distributed Computing | ETL | Machine LearningFlexible work options | In office presence with flexibilityMid-level Full Time4900 Tiedeman Road, Brooklyn, OH, United … R1d ago
-
SVP of Product – Data, Analytics, AI/ML USD 270K-330KAnalytics | Business Intelligence | Data Governance | Data Lakes | Data Privacy401k match | Career development opportunities | Donation matching | Flexible spending account | Health and wellness programExecutive-level Full TimeRemote - USA, United States R1d ago
-
Manager, Applied AI Engineer USD 171K-375KAI/ML | ASR | CRM Integration | Go | GuardrailsHybrid work | Remote work | Work-life balanceSenior-level Full TimeRemote (US), United States R1d ago
-
AI/ML Engineering Manager 2 (Travelbank) USD 119K-140KAPI Design | Agile Development | Artificial Intelligence | Backend Development | Code review401k | Adoption Assistance | Healthcare (Medical Dental Vision) | Life insurance | Long-term disabilityMid-level Full TimeMinneapolis, MN, United States1d ago
-
Capacity modeling | Data Governance | Data Integrity | ERP | Forecasting401k match | Dental insurance | Employee assistance program | Employee scholar program | Flexible spending accountsSenior-level Full TimeUS-GA-MIDLAND-MPC ~ 8801/8987 Macon Rd ~ …1d ago
-
Technical Program Manager – AI & Data Platforms USD 119K-140KCloud Migration | Dependency management | Executive reporting | Financial Forecasting | Issue management401k retirement plan | Healthcare (Medical Dental Vision) | Life insurance | Long-term disability | Paid HolidaysMid-level Full TimeHopkins, MN, United States1d ago