Staff AI Evaluation Lead
Tasks
- Build evaluation pipelines
- Create metric definitions
- Define dataset standards
- Define evaluation methodology
- Define success metrics and communicate impact
- Design end to end evaluation workflows
- Determine evaluation depth by risk
- Drive cross-functional alignment
- Evaluate new models and AI capabilities
- Lead automation and evaluation programs
- Maintain failure taxonomy
- Mentor team members
- Pilot emerging tools and approaches
- Rebaseline evaluations for model changes
- Redesign workflows and tooling for scale
- Set production quality bar
- Validate metrics against production behavior
Perks/Benefits
- N/A
Skills/Tech-stack
API | Automation | Data Pipelines | Data sets | Evaluation | Human-in-the-loop | JSON | LLM | Machine Learning | Python | SQL | Scripting | Systems Design | The Loop
Education
N/A
Roles
AI | AI Engineer | AI Evaluation Lead | Engineer | Evaluation Lead | Lead | Staff AI Engineer
Related jobs
-
AWS | Amazon Redshift | Data Observability | Data Quality | Data ReconciliationInternational team collaboration | Remote work LATAMSenior-level Contract Full TimeGuatemala, Guatemala (Remote) R3h ago
-
AWS | AWS Redshift | AWS Redshift Administration | Amazon Redshift | AutomationRemote workSenior-level Full TimeGuatemala, Guatemala (Remote) R3h ago
-
Agentic Systems | BGP | DNS | Distributed Systems | EnvoyBerlitz tutoring | Catered meals | Flexible learning support | Language tutoring | MentorshipSenior-level Full TimeMontreal, Quebec; Toronto, Ontario; Remote, Canada; … R4h ago
-
Senior Software Engineer, Embedded UI GBP 75K-90KAI | API Design | BrightScript | JavaScript | Language ModelsCommuter benefits | Dental insurance | Disability insurance | Financial wellness support | Health insuranceSenior-level Full TimeManchester, United Kingdom R12h ago
-
Algorithm Engineer USD 150K-170KCI/CD | Code review | Data labeling | Deep learning | Digital SignalIn person office hubs | Remote work flexibilityMid-level Full TimeBoston, MA / Remote R17h ago
-
Access Permissions | Azure Data | Azure Data Factory | Azure DevOps | Azure SynapseExtra vacation days | Flexible work (remote/hybrid) | Health insurance | Internal conferences and meetups | MentorshipMid-level Full TimePoland R19h ago
-
Analytics Engineer USD 130K-170KCI/CD | DBT | Data Modeling | Data Quality | Data Quality Testing401k match | Flexible time off | Health benefits day one | Paid caregiver leave | Paid parental leaveSenior-level Full TimeUnited States (Remote) R19h ago
-
Senior Principal Software Engineer, Machine Learning USD 222K-453KA/B | A/B Testing | B testing | Canary Deployment | Capacity PlanningHybrid work model | MentorshipSenior-level Full TimeRemote, US R20h ago
-
Staff Machine Learning Engineer, Shopping Ads USD 230K-322KConversion modeling | Data labeling | Deep learning | Distributed Systems | Feature Engineering401k employer match | Comprehensive healthcare benefits | Family planning support | Flexible vacation | Gender-affirming careSenior-level Full TimeRemote - United States R20h ago
-
Mid-level Full TimeSingapore R20h ago
-
Data Engineer USD 81K-105KAPIs | AWS | Azure | Cloud Data | Cloud Data Platforms401k match | Full-flex work week | Health and wellness benefits | Hybrid work schedule | Life insuranceMid-level Full TimeUSA VA Falls Church - 3150 … R20h ago
-
Staff Machine Learning Engineer INR 3600K-5000KA/B | A/B Testing | AWS | Agentic Workflows | AzureHybrid work | Onsite collaborationSenior-level Full TimeIndia - Pune - Office R20h ago
-
Analytics Engineer USD 84K-105KAWS | Azure | Cloud Computing | Data Modeling | Data ValidationEmployee assistance program | Health insurance | Paid time off | Retirement plan 401kMid-level Full TimeNew York, NY ( 200 Park … R20h ago
-
Staff ML Engineer, Perception Research USD 251K-310K3D Perception | Autoregressive models | Data Pipelines | Data parallelism | Diffusion Models401k match | Baby bonding leave | Dental insurance | Disability insurance | Health insuranceSenior-level Full TimeMountain View, CA, USA ; New … R21h ago
-
Senior ML Engineer, Perception Research USD 213K-263K3D Perception | Autoregressive models | Camera | Computer Vision | Data Pipelines401k matching | Baby bonding leave | Dental insurance | Disability insurance | Health insuranceSenior-level Full TimeMountain View, CA, USA ; New … R21h ago
-
Director, Embedded Memory EUR 91K-120KAdvanced nodes | Automation | Automation Development | BTI | Bitcell DesignFinancial benefits | Hybrid office location | Industry conference visibility | Medical benefits | Team leadership developmentExecutive-level Full TimeMontbonnot-Saint-Martin, Auvergne-Rhône-Alpes, France R21h ago
-
Software Engineer, Inference Runtime USD 150K-350KC++ | CPU Profiling | CUDA | Executorch | GPU ProfilingDental insurance | Flexible PTO | Flexible WFH | Medical insurance | Team mealsMid-level Full TimeNew York City R21h ago
-
Applied AI/ML Engineer USD 150K-220KAgentic Systems | Cloud Computing | Containerization | Data Storage | Data storage and retrieval401k plan | Flexible PTO | Health insurance | Remote-firstSenior-level Full TimeRemote R21h ago
-
Applied AI Architect, Enterprise Tech USD 240K-315KAPI Integration | Cloud Architecture | Data Science | Enterprise Integration | LLM EvaluationFlexible working hours | Generous vacation | Hybrid work policy | Optional equity donation matching | Parental leaveSenior-level Full TimeSan Francisco, CA | New York … R21h ago
-
Senior Software Engineer, Training & Experimentation USD 180K-250KAPI Development | AWS | Asynchronous processing | Automated testing | Azure401k matching | Company holidays | Company winter break | Dental coverage | Flexible scheduleSenior-level Full TimeLondon, England, United Kingdom; New York, … R22h ago
-
Senior Analytics Engineer CAD 106K-120KAirflow | Cost Optimization | DBT | Data Contracts | Data ModelingBreakfast and lunch catering | Fertility healthcare and family forming support | Gym/Fitness card | Health insurance dental | Home office setup budgetSenior-level Full TimeVancouver, BC R22h ago
-
Staff Software Engineer - Data Platform CAD 135K-169KApache Airflow | BigQuery | CI/CD | Cloud platform | Data CatalogDental insurance | Flexible time off | LinkedIn Learning access | Medical insurance | Paid HolidaysSenior-level Full TimeCanada (Remote) R22h ago
-
Predictive Analytics Consultant USD 69K-101KAWS IAM | AWS Lambda | AWS SageMaker | AWS Step Functions | Amazon CloudWatchSenior-level Full TimeUS Remote R23h ago
-
AI Engineer I USD 90K-119KAPI Integration | Azure | CI/CD | Chroma | Docker401k match | Dental insurance | Holidays | Medical insurance | Personal Time OffEntry-level Full TimeRemote, CA, USA R23h ago
-
Senior Software Engineer - Machine Learning USD 183K-223KApache Flink | Apache Spark | Apache Storm | Data Modeling | Deep learning401k match | Dental insurance | Health insurance | Life insurance | Long-term disabilitySenior-level Full TimeSan Francisco, CA 94158, United States R23h ago