Sr. Applied Scientist, AI Evaluation & Quality Systems
Tasks
- Apply evaluation methodology for generative AI
- Build data quality assurance pipelines
- Conduct root cause analysis for evaluation disagreements
- Design ground truth generation pipelines
- Develop quality control solutions for AI evaluation
- Implement real time validation and monitoring
- Perform anomaly and drift detection
- Prototype validate and ship LLM based pipelines and agents
Perks/Benefits
- N/A
Skills/Tech-stack
Anomaly Detection | Calibration | Data Annotation | Deep learning | Drift Detection | Evaluation methodology | Ground Truth | Ground truth generation | LLM-as-a-Judge | Machine Learning | Machine Learning Pipelines | Meta Evaluation | Production Monitoring | Python | Statistics
Education
Related jobs
-
A/B | A/B Testing | B testing | Causal Inference | ClusteringMid-level Full TimeMountain View, CA, USA; San Francisco, …5h ago
-
Entry-level Full TimeUnited States, Washington, Redmond18h ago
-
Senior Data Scientist USD 119K-144KAWS | Azure | Classification | Clustering | Data Warehousing401k | Paid sick leave | Paid vacation | Safe TimeSenior-level Full TimeCarmel, Indiana, United States; Eagan, Minnesota, …19h ago
-
Lead Data Scientist - Experimentation USD 110K-160KA/B | A/B Testing | AWS | B testing | Causal InferenceEducational Training Programs | Employee assistance program | Health and wellbeing programs | Mentoring | Paid HolidaysSenior-level Full TimeBloomington, IL R20h ago
-
Senior Data and Applied Scientist USD 119K-234KAI Foundry | Azure AI | Azure AI Foundry | Azure Data | Azure Data ExplorerSenior-level Full TimeUnited States, Washington, Redmond20h ago
-
Principal Data Scientist – Supply Chain Operations USD 114K-234KAnalytics Cloud | Cloud Applications | Cloud infrastructure | Dashboards | Data Governance401k matching | Life insurance | Medical/Dental/Vision insurance | Paid Holidays | Paid parental leaveSenior-level Full TimeSanta Clara, CA, United States20h ago
-
Azure | Computer Vision | Data Modeling | Data Pipelines | Data cleaningEntry-level InternshipUnited States, Washington, Redmond; United States, …21h ago
-
Entry-level Full TimeAnnapolis Junction, MD21h ago
-
Senior-level Full TimeAnnapolis Junction, MD21h ago
-
Machine Learning Research Scientist, Evaluations USD 180K-225KBenchmark development | Deep learning | Failure analysis | Fine Tuning | Human FeedbackCommuter stipend | Comprehensive health, dental and vision coverage | Generous PTO | Learning and development stipend | Retirement benefitsEntry-level Full TimeSan Francisco, CA; Seattle, WA; New …22h ago
-
Machine Learning Research Scientist, Post-Training USD 180K-225KDeep learning | Fine Tuning | Human Feedback | Instruction Tuning | Language ModelsCommuter stipend | Health, dental and vision coverage | Learning and development stipend | Paid time off | Retirement benefitsMid-level Full TimeSan Francisco, CA; Seattle, WA; New …22h ago
-
(Senior) Scientist, Computational Neuroscience USD 132K-259KComputational modeling | Data Analysis | Machine Learning | Neural Data | Neural data analysisAnnual incentive program | Healthcare coverage | Retirement benefitsSenior-level Full TimeCambridge, MA USA22h ago
-
Principal, Data/Analytics and Benchmarking USD 129K-186KAI | Benchmarking | Dashboarding | Data Governance | Data ModelingSenior-level Full TimeUnited States R22h ago
-
Lead Data Scientist - Applied AI (P4656) USD 125K-207KAI Agent | AI Agents | AI agent evaluation | Agent Traceability | Agent evaluation401k matching | Dental insurance | Health insurance | Health savings account | Paid time offSenior-level Full TimeCincinnati, OH; Chicago, IL23h ago
-
Staff Data Scientist USD 123K-185KArcGIS | Artificial Intelligence | Automated testing | Bias detection | CI/CD401k | Adoption Assistance | Employee assistance programs | Fitness subsidies | Flexible work arrangementsSenior-level Full TimeAustin,Texas,United States23h ago
-
Data Scientist/Senior Data Scientist USD 136K-183KAgentic Systems | Calculus | Classification | Cloud Native | Distributed ComputingSenior-level Full TimeRedwood City, California, United States1d ago
-
Senior Lead Data Scientist USD 204K-226KCode standards | Data Pipelines | Deep learning | Language Processing | MLOps401k plan | Caregiver referral network | Commuter program | Dental insurance | Disability insuranceSenior-level Full TimeAventura, Florida, United States1d ago
-
Senior Lead Data Scientist USD 204K-226KDeep learning | Language Processing | MLOps | Model Deployment | Model Monitoring401k plan | Commuter program | Dental benefits | Disability insurance | Employee assistance programSenior-level Full TimeBoston, Massachusetts, United States1d ago
-
Senior Lead Data Scientist USD 204K-226KData Manipulation | Deep learning | Language Processing | MLOps | Machine Learning401k plan | Commuter program | Dental insurance | Disability insurance | Employee assistance programSenior-level Full TimeAustin, Texas, United States1d ago
-
Senior Lead Data Scientist USD 204K-226KDeep learning | Language Processing | MLOps | Machine Learning | Model Monitoring401k plan | Commuter program | Dental insurance | Disability insurance | Employee assistance programSenior-level Full TimeWashington, District of Columbia, United States1d ago
-
Senior Lead Data Scientist USD 204K-226KBest practices | Clean Code | Data Analysis | Data Manipulation | Deep learning401k plan | Commuter program | Dental insurance | Disability insurance | Employee assistance programSenior-level Full TimeSeattle, Washington, United States1d ago
-
Senior Data Scientist USD 80KA/B | A/B Testing | B testing | Behavioral Modeling | Business IntelligenceComprehensive benefits | Flexible PTOSenior-level Full TimeUnited States1d ago
-
Principal Applied Scientist for Copilot Evals USD 142K-274KBias | Evaluation | Experimental Design | Experimentation | Information RetrievalSenior-level Full TimeUnited States, Washington, Redmond1d ago
-
Senior Data Scientist for Copilot Evals USD 119K-234KAnomaly Detection | Causal Inference | Classification | Clustering | Data GovernanceSenior-level Full TimeUnited States, Washington, Redmond1d ago
-
Scientist II / Senior Scientist, Computational Biology USD 130K-180KComputational Biology | Data Analysis | Machine Learning | Python | RNA SequencingAnnual incentive program | Healthcare coverage | Retirement benefitsSenior-level Full TimeSomerville, MA1d ago