Applied AI Manager-Evaluation & Measurement
Tasks
- Build evaluation measurement pipelines
- Communicate project status risks and mitigation
- Communicate with customers and stakeholders
- Define evaluation criteria
- Design acceptance tests
- Establish data truth links
- Identify improvement opportunities
- Implement record intake process
- Manage budget cost schedule and ROI
- Monitor AI operational excellence program
- Produce forecast versus actual reports
- Provide AI excellence training
- Reinforce organizational mission values and culture
- Scope auditable evaluation instrument
- Set trace and telemetry standards
Perks/Benefits
- 401k savings plan
- 9 80 program
- Community service hours
- Competitive vacation
- Dental insurance
- Disability insurance
- Flexible work schedules
- Health Emergency Leave
- Health Savings Account contribution
- Health insurance
- Legal insurance
- Life insurance
- Limited benefits eligibility
- Paid parental leave
- Telemedicine
- Vision insurance
- Wellness programs
Skills/Tech-stack
Audit | Baselines | Budget Management | Confidence Bounds | Cost accounting | Dashboards | Data Governance | Data Pipelines | Data measurement | Error Budgeting | Evaluation | Language Model Evaluation | Large Language Model | Large language model evaluation | Machine Learning | Metrics | Model Evaluation | Model Validation | Operational excellence | Probabilistic Acceptance | Project Management | Record Keeping | Risk Management | Statistical Analysis | Telemetry | Telemetry Logging
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Roles
AI | AI Evaluation Manager | AI Manager | Evaluation Manager | Manager
Related jobs
-
Operations Analytics Manager USD 140K-170KBusiness Analytics | Capacity Planning | Data Engineering | Data Governance | Data Modeling401k retirement savings | Equity in high growth startup | Healthcare FSA/HSA subsidy | Medical, dental, and vision coverage | Paid HolidaysMid-level Full TimeRemote, United States R9h ago
-
Product Manager, Robotics & Physical AI USD 140K-176KCalibration | Coordinate systems | Data Annotation | Data Capture | Data PipelinesRegular travel | Remote workMid-level Full TimeRemote, US R10h ago
-
Physical AI Robotics Product Manager USD 140K-176KAgile | Calibration | Computer Vision | Coordinate frames | Data QualityHands on impact in early market | Regular travel to data collection sites | Remote work optionMid-level Full TimeRemote, US R10h ago
-
2027 Technology, Data, AI & Ventures Summer Internship Program - AI Engineer (MLOps) Intern USD 60K-70KAPIs | Automated testing | CI/CD | DevOps | Generative AICapstone project | Hybrid work | Networking opportunitiesEntry-level InternshipNew York, NY, US11h ago
-
Advanced Excel | Business Intelligence | Data Governance | Data Management | Data analytics401k employer match | Commuter benefits | Employee stock purchase program | Employer paid medical/dental/vision | FSAMid-level Full TimeRemote - USA R11h ago
-
Senior AI/LLM Engineer USD 179K-201KAPI Development | CI/CD | Embeddings | Fine Tuning | Guardrails401k match | Dental insurance | Health insurance | Remote work | Vision insuranceSenior-level Full TimeRemote (US/Canada) R12h ago
-
Manager, Data Analytics USD 110K-125KA/B | A/B Testing | Adobe Analytics | Amazon Athena | Amazon QuickSightMid-level Full TimeEl Segundo, CALIFORNIA, United States12h ago
-
API Integration | Agent Frameworks | Artificial Intelligence | Automation | Code AutomationSenior-level Full TimeEl Segundo, CALIFORNIA, United States14h ago
-
Digital Analytics & Engineering Manager USD 85K-114KA/B | A/B Testing | Adobe Analytics | Adobe Experience | Adobe Experience PlatformHybrid workMid-level Full TimeDraper, UT (Hybrid) R14h ago
-
Data Science Manager Provider Product USD 190K-270KCausal Inference | Experimentation | Generative AI | Machine Learning | Python401k employer match | Catered lunch | Cell phone reimbursement | Commuter benefits | Corporate wellness programMid-level Full TimeBoston, MA; New York, NY; Silicon …14h ago
-
AI Solutions Architect USD 150K-185KAPI Integration | AWS | Agent Orchestration | EDI | ERP401k match | Life insurance | Long-term disability | Medical/Dental/Vision insurance | Paid time offSenior-level Full TimeChicago, IL, United States R14h ago
-
Senior Team Leader, AI Engineering USD 200K-280KAgentic Systems | Artificial Intelligence | Evaluation | GraphQL | Human-in-the-loop401k matching | Commuter benefits | Health, dental & vision plans | Lunch daily | Snacks and coffeeSenior-level Full TimeSan Francisco, CA, Washington, D.C., New … R15h ago
-
AWS | Analytics | Architecture Design | Azure | Cloud ArchitectureSenior-level Full TimeRemote - California; Remote - Colorado; … R15h ago
-
Senior Manager, Head of Data & AI USD 140K-170KAI Governance | AI RMF | Access Management | Analytics | Apache SparkHybrid workSenior-level Full TimeLake Mary, FL, US15h ago
-
Development Operations & Analytics Manager USD 70K-90KArtificial Intelligence | Automation | Customer Relationship Management | Customer relationship | DashboardsDental insurance | Early Leave For Shabbat | Medical insurance | Paid sick leave | Paid vacationMid-level Full TimeNew York15h ago
-
Machine Learning Engineering Manager, Personalization USD 184K-262KDeep learning | Engineering leadership | Experimentation | Generative AI | Language Models401k retirement plan | Flexible holidays | Health insurance | Meal allowance | Paid parental leaveMid-level Full TimeNew York, NY16h ago
-
Sr. Engineering Manager, Inference USD 188K-303KAPI Design | API Tracing | Autoscaling | Cloud infrastructure | Compliance401k employer match | Casual work environment | Catered lunch | Company paid life insurance | Employee stock purchase programSenior-level Full TimeLivingston, NJ / New York, NY … R16h ago
-
Staff Product Manager, Embedded USD 280K-350KAPI | API Design | API Integration | Data Modeling | Developer experienceHybrid work | Remote work up to four weeks per yearSenior-level Full TimeNew York, New York, United States R17h ago
-
Staff Product Manager, Embedded USD 280K-350KAPI Design | API Integration | Data Modeling | Developer experience | Distributed SystemsHybrid work environment | Remote work up to four weeks per yearSenior-level Full TimeSeattle, Washington, United States R17h ago
-
AI Engineer — GTM Analytics USD 146K-201KAgents | CI/CD | Databricks Jobs | Delta Lake | EvaluationSenior-level Full TimeUnited States17h ago
-
Sr. Manager, People Data Governance USD 148K-243KAccess Control | Business glossary | Data Catalog | Data Governance | Data LineageCareer development | Hybrid work scheduleSenior-level Full TimeSunnyvale, CA, United States19h ago
-
Senior Manager, Data Science USD 187K-217KAgile | Airflow | DBT | Data Science | Data VisualizationSenior-level Full TimeCA - San Francisco; NY - …19h ago
-
AI Systems Engineer-Senior USD 140K-206KAWS Lambda | Agentic AI | Amazon EC2 | Amazon ECS | Amazon EKSSenior-level Full TimeAnnapolis Junction, Maryland, United States20h ago
-
AI/ML Engineer USD 108K-203KAmazon SageMaker | Automated retraining | Azure ML | Azure Machine Learning | CI/CDSenior-level Full TimeWashington, DC20h ago
-
Manager, AI Engineering USD 49KArtificial Intelligence | CI/CD | Computer Vision | Deep learning | DockerMid-level Full TimeMorrisville, North Carolina, États-Unis d’Amérique21h ago