Staff AI Evaluation Lead
Tasks
- Build evaluation pipelines
- Create metric definitions
- Define dataset standards
- Define evaluation methodology
- Define success metrics and communicate impact
- Design end to end evaluation workflows
- Determine evaluation depth by risk
- Drive cross-functional alignment
- Evaluate new models and AI capabilities
- Lead automation and evaluation programs
- Maintain failure taxonomy
- Mentor team members
- Pilot emerging tools and approaches
- Rebaseline evaluations for model changes
- Redesign workflows and tooling for scale
- Set production quality bar
- Validate metrics against production behavior
Perks/Benefits
- N/A
Skills/Tech-stack
API | Automation | Data Pipelines | Data sets | Evaluation | Human-in-the-loop | JSON | LLM | Machine Learning | Python | SQL | Scripting | Systems Design | The Loop
Education
N/A
Roles
AI | AI Engineer | AI Evaluation Lead | Engineer | Evaluation Lead | Lead | Staff AI Engineer
Related jobs
-
AI Evaluation | CI/CD | Cloud | DevOps | LLMPaid in USD | Remote workSenior-level Full TimeGuatemala, Guatemala (Remote) R1d ago
-
AC DC Sizing | AC/DC | Arc Flash | Artificial Intelligence | AutomationMultiple project extensions | Paid on time | Remote work optionSenior-level Full TimeJuno Beach, FL R1d ago
-
AI Agents | API Development | Azure | CI/CD | LLMOpsCollaborative cross-functional team | Fully remoteMid-level Full TimeCanada, Ontario; Toronto, None; Toronto, Ontario; … R1d ago
-
Benchmarking | C plus plus | CUDA | Computational Engineering | Distributed Computing401k match | Generous paid time off | Medical, dental & vision coverage | Stock options | WFH Hardware ReimbursementSenior-level Full TimeSunnyvale, CA R1d ago
-
Staff ML Engineer, Foundation Models USD 251K-310KBayesian Inference | Foundation Models | Generative Modeling | Hierarchical learning | Language ModelsBonus | Equity incentiveSenior-level Full TimeMountain View, CA, USA R1d ago
-
AI Agents | AI Governance | API Security | Auditability | Cloud SecurityHybrid work | Onsite supportMid-level Full TimeBengaluru, Karnataka, India R1d ago
-
AI/ML Engineering Manager INR 1900K-2800KAWS CDK | AWS Lambda | AWS SageMaker | Amazon ECS | Amazon EKSAccess to Headspace app | Flexible vacation | Hybrid work model | Mental health days | Mental physical and financial wellbeing resourcesMid-level Full TimeIndia, Bengaluru, Karnataka R1d ago
-
API Gateway | AWS Lambda | AWS Step Functions | Amazon SQS | Apache Airflow401k matching | Employee stock options | Health benefits | Hybrid work onsite 2 to 3 days per week | Paid time offExecutive-level Full TimeNew York, United States R1d ago
-
AIOps | Anomaly Detection | Anthos | CRD | Control Plane401k plan | Basic life insurance | Life insurance | Long-term disability | Medical/Dental/Vision insuranceSenior-level Full TimeResearch Triangle Park, North Carolina, United … R1d ago
-
Senior Machine Learning Engineer USD 160K-200KAWS | CI/CD | Computer Vision | Data Processing | DockerCommuter benefits | Dental insurance | Employee assistance program | Equity | Health insuranceSenior-level Full TimeArlington, VA R1d ago
-
Senior Computer Vision & Machine Learning Engineer USD 140K-206KAblation Studies | CI/CD | Data Pipelines | Docker | Error AnalysisAutonomy | Client project impact | Support for professional growthSenior-level Full TimeRemote, US R1d ago
-
Computer Vision & Machine Learning Engineer USD 150K-200KAblation Studies | CI/CD | Computer Vision | Data Pipelines | DockerEntry-level Full TimeRemote, US R1d ago
-
Data Engineer | CDMX | Híbrido MXN 480K-480KAgile | Amazon EMR | Amazon Redshift | Amazon Web Services | Apache AirflowGrowth plan | International exposure | Legal benefits | Training and certificationsEntry-level Full TimeMexico City, Mexico City, Mexico R1d ago
-
Staff Data Engineer USD 162K-224KAmazon Managed Workflows for Apache Airflow | Apache Airflow | Apache Druid | Apache Iceberg | CCPA401k employer match | Annual WIN bonus | Flexible vacation | Mental health resources | Paid HolidaysSenior-level Full TimeU.S. Remote R1d ago
-
Sr Staff AI Engineer - Veza USD 190K-334KAgent Orchestration | Algorithms | C++ | Data Structures | Distributed Systems401k plan with match | ESPP | Family leave programs | Flexible time away | Health plansSenior-level Full TimeMinneapolis, Minnesota, United States R1d ago
-
AI/ML Engineer USD 152K-220K3D Reconstruction | 3D Scene | 3D Scene Understanding | AI Agents | Cloud ComputingAgile team collaboration | Fully remote | Production monitoring ownershipMid-level Full TimeSan Francisco, CA (Hybrid) or United … R1d ago
-
Senior GTM AI Engineer USD 154K-213KAgent Orchestration | Agent Performance | Agent Performance Monitoring | Anthropic Claude | App Development401k match | Catered lunch | Company events | Desk setup stipend | Flexible spending accountsSenior-level Full TimeOakland, California, United States R1d ago
-
Lead Applied AI and Data Scientist | IFS Copperleaf CAD 113K-152KA/B | A/B Testing | Anomaly Detection | B testing | Causal InferenceSenior-level Full TimeVancouver, British Columbia, Canada R1d ago
-
Ingeniero Big Data MXN 480K-480KArtifactory | CI/CD | Cloudera | ETL | GitHybrid work | Technical documentationSenior-level Full TimeCiudad de México, CDMX, Mexico R1d ago
-
Entry-level Full TimeBrasil R1d ago
-
Machine Learning Engineer CAD 110K-130KAmazon Web Services | C++ | Cloud platform | Code Quality | Computer VisionEmployee wellness programs | Free on-site gym | Hybrid work option | Modern office | On-site daycareMid-level TemporaryToronto, Ontario, CA R1d ago
-
Data Science Engineer USD 121K-154KData Visualization | Documentation | Entity Resolution | Fine Tuning | Git401k | Education reimbursement program | Flexible schedules | Hybrid schedule | Relocation assistanceEntry-level Full TimeLivermore, CA, United States R1d ago
-
Senior Data Engineer CZK 1092K-1500KAWS | Airflow | Amazon Redshift | Apache Spark | CI/CDCoursera access | Employee referral rewards | Flexible time off | Paid public holidays | Paid sick leaveSenior-level Full TimeRemote - Czech Republic R1d ago
-
Software Engineer, Applied AI Research USD 180K-400KBackend architecture | Generative AI | Image Generation | Language Models | Large Language ModelsCareer growth potential | Remote-first work environmentSenior-level Full TimeRemote (North America) R1d ago
-
AI Architect USD 120K-197KAI orchestration | Automation | Cloud Computing | Data Governance | Distributed SystemsWeekly earned wage access optionSenior-level Full TimeDurant, OK, United States R1d ago