Principal Applied Scientist for Copilot Evals
United States, Washington, Redmond
USD 142K-274K Senior-level Full Time
Tasks
- Assess evaluation alignment with production outcomes
- Attribute quality losses across grounding retrieval tools orchestration model reasoning and response generation
- Convert evidence into reusable evaluation and reward signals
- Design behavior and task evaluations rubrics graders and calibration
- Develop methods for evaluation set representativeness and coverage
- Mentor scientists and engineers
- Set scientific strategy for customer grounded quality
- Translate user research and customer feedback into evaluation priorities
Perks/Benefits
- N/A
Skills/Tech-stack
Bias | Evaluation | Experimental Design | Experimentation | Information Retrieval | Language Models | Language Processing | Large Language Models | Machine Learning | Measurement Validity | Natural Language | Natural Language Processing | Reinforcement Learning | Retrieval-Augmented Generation | Reward Functions | Sampling | Statistical inference | Statistics | Uncertainty
Education
Regions
Countries
States
Related jobs
-
A/B | A/B Testing | B testing | Causal Inference | ClusteringMid-level Full TimeMountain View, CA, USA; San Francisco, …5h ago
-
API performance | Agent systems | AlloyDB AI | BigQuery | C++Senior-level Full TimeAtlanta, GA, USA; Austin, TX, USA5h ago
-
Entry-level Full TimeUnited States, Washington, Redmond17h ago
-
Senior Data Scientist USD 119K-144KAWS | Azure | Classification | Clustering | Data Warehousing401k | Paid sick leave | Paid vacation | Safe TimeSenior-level Full TimeCarmel, Indiana, United States; Eagan, Minnesota, …18h ago
-
Sr. Applied Scientist, AI Evaluation & Quality Systems USD 170K-300KAnomaly Detection | Calibration | Data Annotation | Deep learning | Drift DetectionSenior-level Full TimeSeattle18h ago
-
Lead Data Scientist - Experimentation USD 110K-160KA/B | A/B Testing | AWS | B testing | Causal InferenceEducational Training Programs | Employee assistance program | Health and wellbeing programs | Mentoring | Paid HolidaysSenior-level Full TimeBloomington, IL R20h ago
-
Senior Data and Applied Scientist USD 119K-234KAI Foundry | Azure AI | Azure AI Foundry | Azure Data | Azure Data ExplorerSenior-level Full TimeUnited States, Washington, Redmond20h ago
-
Principal AI Engineer II USD 148K-258KAPIs | AWS | Artificial Intelligence | CI/CD | Change Control401k | Dental insurance | Medical insurance | Paid time off | Vision insuranceSenior-level Full TimeNorth Chicago, IL, United States20h ago
-
Principal Data Scientist – Supply Chain Operations USD 114K-234KAnalytics Cloud | Cloud Applications | Cloud infrastructure | Dashboards | Data Governance401k matching | Life insurance | Medical/Dental/Vision insurance | Paid Holidays | Paid parental leaveSenior-level Full TimeSanta Clara, CA, United States20h ago
-
Azure | Computer Vision | Data Modeling | Data Pipelines | Data cleaningEntry-level InternshipUnited States, Washington, Redmond; United States, …21h ago
-
Entry-level Full TimeAnnapolis Junction, MD21h ago
-
Senior-level Full TimeAnnapolis Junction, MD21h ago
-
Machine Learning Research Scientist, Evaluations USD 180K-225KBenchmark development | Deep learning | Failure analysis | Fine Tuning | Human FeedbackCommuter stipend | Comprehensive health, dental and vision coverage | Generous PTO | Learning and development stipend | Retirement benefitsEntry-level Full TimeSan Francisco, CA; Seattle, WA; New …21h ago
-
Machine Learning Research Scientist, Post-Training USD 180K-225KDeep learning | Fine Tuning | Human Feedback | Instruction Tuning | Language ModelsCommuter stipend | Health, dental and vision coverage | Learning and development stipend | Paid time off | Retirement benefitsMid-level Full TimeSan Francisco, CA; Seattle, WA; New …21h ago
-
(Senior) Scientist, Computational Neuroscience USD 132K-259KComputational modeling | Data Analysis | Machine Learning | Neural Data | Neural data analysisAnnual incentive program | Healthcare coverage | Retirement benefitsSenior-level Full TimeCambridge, MA USA22h ago
-
Quantum Physicist, Integrated Photonics USD 140K-175KCAD | Computer Aided Design | Custom Electronics Design | Custom electronics | Electronics designMid-level Full TimeMalibu, CA22h ago
-
Lead Data Scientist - Applied AI (P4656) USD 125K-207KAI Agent | AI Agents | AI agent evaluation | Agent Traceability | Agent evaluation401k matching | Dental insurance | Health insurance | Health savings account | Paid time offSenior-level Full TimeCincinnati, OH; Chicago, IL23h ago
-
Staff Data Scientist USD 123K-185KArcGIS | Artificial Intelligence | Automated testing | Bias detection | CI/CD401k | Adoption Assistance | Employee assistance programs | Fitness subsidies | Flexible work arrangementsSenior-level Full TimeAustin,Texas,United States23h ago
-
Data Scientist/Senior Data Scientist USD 136K-183KAgentic Systems | Calculus | Classification | Cloud Native | Distributed ComputingSenior-level Full TimeRedwood City, California, United States23h ago
-
Principal Product Manager, Experimentation Platform USD 187K-260KData Science | Experiment design | Feature Flags | Machine Learning | Metric Design401k match | Dental insurance | Disability coverage | Employee assistance program | Employee resource groupsSenior-level Full TimeUnited States | Remote R23h ago
-
Senior Lead Data Scientist USD 204K-226KCode standards | Data Pipelines | Deep learning | Language Processing | MLOps401k plan | Caregiver referral network | Commuter program | Dental insurance | Disability insuranceSenior-level Full TimeAventura, Florida, United States1d ago
-
Senior Lead Data Scientist USD 204K-226KDeep learning | Language Processing | MLOps | Model Deployment | Model Monitoring401k plan | Commuter program | Dental benefits | Disability insurance | Employee assistance programSenior-level Full TimeBoston, Massachusetts, United States1d ago
-
Senior Lead Data Scientist USD 204K-226KData Manipulation | Deep learning | Language Processing | MLOps | Machine Learning401k plan | Commuter program | Dental insurance | Disability insurance | Employee assistance programSenior-level Full TimeAustin, Texas, United States1d ago
-
Senior Lead Data Scientist USD 204K-226KDeep learning | Language Processing | MLOps | Machine Learning | Model Monitoring401k plan | Commuter program | Dental insurance | Disability insurance | Employee assistance programSenior-level Full TimeWashington, District of Columbia, United States1d ago
-
Senior Lead Data Scientist USD 204K-226KBest practices | Clean Code | Data Analysis | Data Manipulation | Deep learning401k plan | Commuter program | Dental insurance | Disability insurance | Employee assistance programSenior-level Full TimeSeattle, Washington, United States1d ago