Technical Lead - Agent & Model Evaluation
Tasks
- Build adversarial evaluation harnesses
- Build automated evaluation pipelines
- Create automated RL and fine-tuning pipelines
- Create benchmark suites for agent capabilities
- Define metrics for agent evaluation
- Design evaluation framework
- Develop failure analysis systems
- Develop replay systems for agent improvement
- Establish quality gates and release criteria
- Implement A/B testing and deployment infrastructure
Perks/Benefits
Skills/Tech-stack
A/B | A/B Testing | Automation | B testing | Benchmarking | Deep learning | Evaluation systems | ML pipelines | Machine Learning | Reinforcement Learning | Statistical Analysis | Systems engineering
Education
Roles
Regions
Countries
States
Cities
Related jobs
-
Lead Data Architect USD 173K-276KAI metadata | Cloud Platforms | Data Architecture | Data Governance | Data Lakes401k | Dental insurance | Health insurance | Paid time off | Vision insuranceExecutive-level Full TimeFoster City, CA, United States4h ago
-
Principal System Integration Lead (Quantum Hardware) USD 195K-260KCause analysis | Change Control | Configuration Management | Controls engineering | DOORSSenior-level Full TimeMilpitas, California, United States16h ago
-
Lead Machine Learning Engineer USD 157K-237KA/B | A/B Testing | Airflow | B testing | Data PipelinesSenior-level Full TimeUS TX Austin18h ago
-
Lead Data Scientist USD 225K-275KA/B | A/B Testing | API Integration | B testing | DBTCollaborative work environment | Dental insurance | Education Impact | Health insurance | Hybrid work modelSenior-level Full TimeSan Francisco R19h ago
-
Technical Lead Manager, AI/ML Networking USD 207K-300KArtificial Intelligence | C++ | Compute Technologies | Dataplane Encryption | Deep Learning Execution ProviderSenior-level Full TimeRaleigh, NC, USA; Durham, NC, USA1d ago
-
Autonomy | Closed Loop | Closed loop control | Computer Vision | Control SystemsSenior-level Full TimeNew York, New York, USA1d ago
-
Senior-level Full TimePortland, Oregon, United States1d ago
-
AI workflow | AI workflow design | Artificial Intelligence | Business Communication | Data ScienceSenior-level Full TimeNorth Carolina, North Carolina, United States1d ago
-
Data Science Manager, Borrow USD 156K-179KAirflow | DBT | Data Pipelines | Data Visualization | ExperimentationMid-level Full TimeCA - San Francisco; NY - …1d ago
-
Senior-level Full TimeChicago, Illinois, USA R2d ago
-
AWS | Agile | Amazon Web Services | Apache Spark | Data EngineeringAccess to cutting-edge technologies | Collaborative team environment | Flexible work hours | Professional development opportunities | Remote work within United StatesSenior-level Full TimeMassachusetts R2d ago
-
Agile | Artificial Intelligence | CAP | CLIA | Clinical software401k match | Career coaching | Employee resource groups | Employee stock purchase program | Leadership developmentSenior-level Full TimePennsylvania R2d ago
-
Lead Data Analyst USD 147K-170KAWS | CI/CD | Cloud Composer | Cloud Functions | Cloud RunDental insurance | Health care | Paid time off | Retirement plan | Sick leaveSenior-level Full TimeSt. Louis, Missouri, US R2d ago
-
Workforce Analytics Lead USD 105K-166KAccess | Data Governance | Data Management | Data Visualization | Excel401k plan | 401k retirement plan match | Dental insurance | Group health benefits | Hybrid work environmentSenior-level Full TimeAlexandria, VA, United States2d ago
-
Lead Data Scientist (Predictive Analytics) USD 111K-180KData Modeling | Data Visualization | Excel | Forecasting | Machine LearningSenior-level Full TimeAlexandria, VA, United States2d ago
-
Lead Data Scientist USD 111K-180KAdobe Acrobat | Business Intelligence | Data Modeling | Data analytics | KPI designSenior-level Full TimeAlexandria, VA, United States2d ago
-
Lead Software Engineer - AI Infrastructure & Tooling USD 155K-218KAI/ML | APIs | AWS | Agile | Amazon KinesisSenior-level Full TimeUSA - CA - 2450 Broadway, …2d ago
-
Quantitative Analytics Tech Lead USD 144K-216KAlgorithms | Bayesian Methods | Bureau data | Credit Bureau Data | Credit RiskSenior-level Full TimeWestbranch, United States2d ago
-
Business Analytics Lead USD 74K-161KBusiness Analytics | Business Intelligence | Data Quality | Data integration | Data-Driven Decision Making401k match | Dental insurance | Disability insurance | Educational assistance | Health insuranceSenior-level Full TimeDenver - 12345 W Colfax (CO147), …2d ago
-
Lead Data Engineer USD 129K-161KAutomation | Azure Data | Azure Data Factory | Azure Purview | BI reportingMental health services | Paid parental leave | Paid sick days | Paid time off | Paid vacationSenior-level Full TimeChicago - Illinois - Wacker, United …2d ago
-
Lead Data Scientist USD 150K-175KCloud Computing | Clustering | Computer Vision | Data Streaming | Deep learning401k plan with company matching | Employee assistance program | Health and wellbeing | Holidays | Life and disability insuranceSenior-level Full TimeRemote - Nationwide, United States R2d ago
-
Lead Software Engineer-Python/Data Engineering USD 172K-215KAWS ECS | AWS EMR | AWS Lambda | Agile | Amazon EC2Backup childcare | Financial coaching | Health care coverage | Mental health support | On-site health and wellness centersSenior-level Full TimeWilmington, DE, United States3d ago
-
Lead Data Analyst - On-site in Jackson, MI USD 117K-174KClustering | Data Modeling | Databricks | Ensemble Methods | Machine Learning401k retirement plan | Dental insurance | Flexible time off | Medical insurance | Paid HolidaysSenior-level Full TimeDetroit, United States3d ago
-
BigQuery | CI/CD | Containerization | Data Modeling | DatabricksSenior-level ContractNew York, NY (Hybrid) R3d ago
-
AI/ML Deployment and Enablement Lead USD 170K-200KAWS | Amazon Bedrock | Apache Airflow | Artificial Intelligence | Auditability401k match | Adoption Assistance | Healthcare coverage | Long-term disability | Paid HolidaysSenior-level Full TimeMinneapolis, MN, United States3d ago