AIML - Sr Machine Learning Engineering Manager, Evaluation
Tasks
- Build automated prompt optimization systems
- Conduct architecture and code reviews
- Design evaluation pipelines for offline benchmarks and simulation
- Diagnose model failure modes
- Drive improvements to prompts context tools rubrics and agent harnesses
- Ensure data quality representativeness privacy and reproducibility
- Evaluate foundation models and agents
- Generate and evaluate synthetic datasets
- Integrate evaluation with post training data generation
- Lead technical strategy for agent evaluation
- Mentor engineers and grow technical talent
- Prototype evaluation and optimization approaches
- Translate research into scalable evaluation systems
Perks/Benefits
- N/A
Skills/Tech-stack
Agentic Systems | Automatic Prompt Optimization | Context Optimization | Data Generation | Data Privacy | Data Quality | Deduplication | Deep learning | Device Machine Learning | Evaluation Benchmark | Experimentation platform | Filtering | Human Feedback | LLM-based Judge | Language Models | Language Processing | Large Language Models | Learning from Human Feedback | ML Pipeline | Machine Learning | Multi-Objective Optimization | Natural Language | Natural Language Processing | Objective optimization | On-device Machine Learning | Prompt Optimization | Python | Reinforcement Learning | Reinforcement Learning from Human Feedback | Reward Modeling | Rubric development | Simulation | Synthetic Data Generation | Synthetic data | Tool use
Education
Regions
Countries
States
Cities
Related jobs
-
Senior Manager, Data Science - Styling Algorithms USD 200K-246KA/B | A/B Testing | Agentic AI | Artificial Intelligence | B testingAnnual bonus | Dental insurance | Equity grants | Health insurance | Restricted stock unitsSenior-level Full TimeRemote USA R1d ago
-
Senior-level Full TimeMiami, United States R1d ago
-
AI Solutions Deployment Manager, Cloud Applied AI, Telco USD 183K-265KAgentic AI | Cloud Architecture | Cloud platform | Conversational AI | Evaluation FrameworksSenior-level Full TimeWaterloo, ON, Canada; New York, NY, …1d ago
-
Director, Product Management, Cloud AI and Science USD 281K-391KAI Agents | Agent systems | Agentic Workflows | Artificial Intelligence | Business StrategyHealth and wellness benefits | Paid time off | Retirement savings planExecutive-level Full TimeSunnyvale, CA, USA; New York, NY, …1d ago
-
Manager, Technical Solutions, Cloud Applied AI USD 233K-324KAgentic AI | C++ | Cloud Architecture | Cloud platform | Conversational AISenior-level Full TimeWaterloo, ON, Canada; Kirkland, WA, USA1d ago
-
Senior Product Manager, Enterprise Data Governance USD 192K-278KAI | Data Domain | Data Governance | Data Literacy | Data PipelinesBonus | Health insurance | Paid time off | Retirement planSenior-level Full TimeSan Jose, CA, USA1d ago
-
Cross-functional | Cross-functional leadership | Data Processing | Debugging | Distributed ComputingSenior-level Full TimeSunnyvale, CA, USA1d ago
-
Analytics Manager GBP 55K-65KDashboard Development | Data Governance | Data Quality | Data Visualization | KPIFlexible benefits | Healthcare benefits | On-target bonus | Paid Holidays | Pension planMid-level Full TimeAny of our offices R1d ago
-
Senior Analytics Manager GBP 63K-80KAI | Analytics | Business Intelligence | Data Governance | Data MaturityCritical illness cover | Dental insurance | Discounts | Extra holiday buying | Flexibility to work from homeSenior-level Full TimeAny of our offices R1d ago
-
Manager, Data Engineering USD 200K-335KApache Airflow | Apache NiFi | Apache Spark | BigQuery | Cloud platformSenior-level Full TimeUS - San Francisco R1d ago
-
COBIT | Data Governance | Data Management | Data Quality | Data SecurityExecutive-level Full TimeOmaha, NE, United States1d ago
-
Atlan | Collibra | Data Cataloging | Data Classification | Data ComplianceSenior-level Full TimeOmaha, NE, United States1d ago
-
Engineering Manager, Data Transformation USD 214K-321KAirflow | Amazon S3 | Flink SQL | GRPC | HiveMid-level Full TimeUS-SEA, US-SF, US-NYC, US-Remote R1d ago
-
Manager, Machine Learning Engineering USD 219K-329KAPIs | AWS | Apache Spark | CI/CD | Code review401k matching | Dental benefits | Employee recognition programs | Flexible time off | Healthcare benefitsSenior-level Full TimeSan Francisco, CA R1d ago
-
Associate Director of Statistical Programming USD 100K-146KADaM | AI integration | Bash | CAPA | CDISCContinuing education | Hybrid work environment | Professional meeting attendance | Work-life balanceMid-level Full TimeBoston, MA, United States R2d ago
-
Lead Enterprise Data Steward USD 61K-83KBusiness glossary | Cause analysis | Data Catalog | Data Governance | Data Governance TechnologiesSenior-level Full TimeReading, PA, United States2d ago
-
Engineering Manager, Data Platform USD 290K-330KAWS | Apache Airflow | Apache Spark | DBT | Data Ingestion401k | Commuter benefits | Employee assistance program | Free Lunches | Medical/Dental/Vision insuranceMid-level Full TimeSan Francisco, CA R2d ago
-
Manager, Stars Data & Analytics USD 94K-158KActive Listening | Application design | Architecture | Budget Management | Data ModelingMid-level Full TimeDetroit, MI, United States2d ago
-
Senior Quantum Chemistry Product Manager USD 176K-207KArtificial Intelligence | Computational Chemistry | Go-to-market | Machine Learning | Product Requirements401k match | Disability insurance | Equity grants | Health insurance | Life insuranceSenior-level Full TimePalo Alto, California, United States R2d ago
-
Technical Program Manager USD 140K-170KAI Agents | API Integration | Data Architecture | Data Governance | Data Leakage PreventionMid-level Full TimeRemote, United States R2d ago
-
AVP Data Science USD 175K-254KClustering | Component analysis | Data Analysis | Data Monitoring | Data Warehouse401k matching | Bonding Leave | Community service pay | Company holidays | Employee auto discountExecutive-level Full TimeFort Worth, TX, United States2d ago
-
Associate Director, International Market Analytics & Business Insights, Immunology (Rheumatology) USD 137K-230KBusiness Intelligence | Competitive Analysis | Cross-Functional Collaboration | Cross-functional | Data Mining401k | Dental insurance | Health insurance | Long-term incentive program | Paid time offMid-level Full TimeMettawa, IL, United States2d ago
-
Claims analytics | Commercial Analytics | Coverage Analytics | Dashboard Development | Data StorytellingExecutive-level Full TimePrinceton, NJ, United States2d ago
-
Technical Product Manager USD 80K-100KA/B Testing | B testing | Cost Optimization | Experiment design | ExperimentationMid-level Full TimeUnited States R2d ago
-
Senior Manager of Data Engineer USD 162K-180KAWS | DBT | Data Governance | Data Quality | Data Security12-week paid parental leave | 4 week paid sabbatical | 401k match | Flexible PTO | Health benefitsSenior-level Full TimeRemote, US R2d ago