Technical Lead - Agent & Model Evaluation
Tasks
- Build adversarial evaluation harnesses
- Build automated evaluation pipelines
- Create automated RL and fine-tuning pipelines
- Create benchmark suites for agent capabilities
- Define metrics for agent evaluation
- Design evaluation framework
- Develop failure analysis systems
- Develop replay systems for agent improvement
- Establish quality gates and release criteria
- Implement A/B testing and deployment infrastructure
Perks/Benefits
Skills/Tech-stack
A/B | A/B Testing | Automation | B testing | Benchmarking | Deep learning | Evaluation systems | ML pipelines | Machine Learning | Reinforcement Learning | Statistical Analysis | Systems engineering
Education
Roles
Regions
Countries
States
Cities
Related jobs
-
Lead Software Engineer USD 179K-215KAWS | AWS CloudWatch | AWS Glue | AWS Lambda | AWS RedshiftBackup childcare | Financial coaching | Health care coverage | Mental health support | On-site wellness centersSenior-level Full TimeJersey City, NJ, United States21h ago
-
Applied AI ML Lead - Java / Python USD 177K-205KAI | AWS | Application Resiliency | Azure | Cloud ComputingFinancial coaching | Health care coverage | Mental health support | On-site wellness centers | Retirement planSenior-level Full TimeJersey City, NJ, United States1d ago
-
Tech Lead Manager ML Optimization USD 298K-378KDeepSpeed | Distributed Training | Hardware optimization | JAX | Low LatencyCompany benefits | Discretionary bonus | Equity incentive | Flexible work arrangementsSenior-level Full TimeMountain View, California1d ago
-
Accelerator | CUDA | GPU | Hardware-Aware Machine Learning | Kernel programmingHybrid work | Relocation assistanceSenior-level Full TimeSan Francisco1d ago
-
Founding Staff ML Engineer (Tech Lead) USD 190K-260KData Processing | Deep learning | Fine Tuning | LLM | ML deploymentSenior-level Full TimePalo Alto, USA Remote, New York R1d ago
-
Tech Lead Manager, Machine Learning Frameworks USD 207K-300KData Processing | Debugging | Fine Tuning | GPU | JAXFlexible work arrangements | Health insurance | Professional developmentSenior-level Full TimeSunnyvale, CA, USA1d ago
-
Lead AI Engineer USD 176K-264KAWS | Alembic | Async Processing | Celery | Distributed SystemsFlexible work environment | Generous leave policy | Health benefitsSenior-level Full TimePalo Alto, CA1d ago
-
AI Data Intelligence Lead USD 187K-328KAI platforms | Data Architecture | Data Engineering | Data Enrichment | Data LineageSenior-level Full TimeUSA - Remote R1d ago
-
AI Transformation Lead USD 140K-190KAI Governance | AI Safety | AI tools | CI/CD | Change ManagementCareer growth programs | Training opportunitiesSenior-level Full TimeMcLean, VA, United States1d ago
-
Lead AI Solutions Engineer - New Business USD 130K-145KAI Development | AI Pipelines | AI toolchains | Automation | Autonomous SystemsFlexible work | Health insurance | Paid time off | Performance bonuses | Retirement planSenior-level Full TimeAustin, TX, USA2d ago
-
Lead AI Solutions Engineer - Customer Experience USD 130K-145KAI Native Solutions | AI-native | API Design | AWS | Artificial IntelligenceCompetitive salary | Flexible work options | Health insurance | Paid time off | Paid volunteer timeSenior-level Full TimeAustin, TX, USA2d ago
-
AI systems | Code review | Communication | Data Pipelines | DeploymentEquity | Flexible work scheduleSenior-level Full TimeSan Francisco, CA2d ago
-
Vice President Data Scientist Lead - Customer Analytics USD 162K-205KA/B | A/B Testing | B testing | Data Management | Data StorytellingHealth care | Mental health support | On-site wellness | Retirement plan | Tuition reimbursementSenior-level Full TimeOH, United States2d ago
-
AI | Automation | Cloud Computing | Data Infrastructure | LeadershipCareer development | Collaborative environment | Inclusive culture | Remote workSenior-level Full TimeFlorida R2d ago
-
AI Technology Enablement Lead USD 135K-216KArtificial Intelligence | Change Management | Cloud Platforms | Cybersecurity | Data ManagementFlexible work arrangements | Training and development opportunitiesSenior-level Full TimeSan Diego, CA, United States2d ago
-
Data Analyst Lead USD 90K-131KBusiness Intelligence | Data Analysis | Data Governance | Data Management | Data VisualizationFlexible schedule | Health benefitsSenior-level Full TimeDallas - Hospital, United States3d ago
-
Analytics Solutions Lead - Pharma USD 142K-284KBusiness Analysis | Data Science | Data Visualization | Excel | Healthcare Analytics401k plan | Counseling services | Dependent care resources | Employee stock purchase plan | Family leaveSenior-level Full TimeNew York-161 Ave of the Americas, …3d ago
-
Senior Manager, AI and Data Scientist USD 150K-224KAI Model Deployment | AI model | AI/ML | AWS SageMaker | Data Architecture401k match | Dental insurance | Flexible time off | Medical insurance | Paid HolidaysSenior-level Full TimePrinceton, United States3d ago
-
Director, Lead Data Scientist (Hybrid) USD 158K-238K6sense | AI | Azure | Bombora | Data Mining401k | Dental | Disability insurance | Education benefits | Employee stock purchaseSenior-level Full TimeWash, 213 Washington St., Newark, NJ, …3d ago
-
Customer Insights, Lead, Quantitative Research USD 123K-202KAI tools | Customer Insights | Data Analysis | Data Visualization | Machine LearningDental insurance | Disability insurance | Life insurance | Medical insurance | Paid time offSenior-level Full TimeNew York - New York, United …3d ago
-
Senior AI Engineering Manager, Ads AI USD 198K-326KAI modeling | Data Systems | Deep learning | Fine Tuning | LLMsHealth and wellness programs | Inclusive culture | Time offExecutive-level Full TimeMountain View, CA, United States3d ago
-
Staff Tech Lead Manager, ML Data Services USD 225K-275KBeam | Cloud Platforms | Data Engineering | Data Governance | Data Pipelines401k | Dental insurance | Health saving accounts | Life insurance | Medical insuranceSenior-level Full TimePittsburgh, Pennsylvania, United States3d ago
-
Staff Tech Lead Manager, ML Data Services USD 225K-275KAWS | Azure | Beam | Cloud Platforms | Data Engineering401k | Dental insurance | Health saving accounts | Life insurance | Medical insuranceSenior-level Full TimeBoston, Massachusetts, United States3d ago
-
Staff Tech Lead Manager, ML Data Services USD 225K-275KAWS | Azure | Beam | Data Governance | Data Infrastructure401k | Dental | Health savings accounts | Life insurance | MedicalSenior-level Full TimeBoston, Massachusetts, United States; Pittsburgh, Pennsylvania, … R3d ago
-
AML Models Analytics Lead USD 129K-194KControl development | Data Analysis | Machine Learning | Model Validation | PythonSenior-level Full TimeLos Angeles, CA | Austin, TX …3d ago