Senior Data Scientist (AI Evaluation & Improvement)
Tasks
- Define evaluation metrics and quality bar
- Design experiments to isolate failure modes
- Document diagnostic playbook
- Extend evaluation platform with golden sets
- Handle clinical data according to privacy compliance
- Mentor others in evaluation methods
- Own LLM diagnostic loop
- Partner with clinical reviewers to label evidence
- Partner with product managers to prioritize failure modes
- Run structured evaluation suites
- Support clinical guidelines regression evaluation
- Test prompt and configuration fixes
Perks/Benefits
Skills/Tech-stack
A/B | A/B Testing | B testing | Cause analysis | Data Analysis | Error Analysis | Experiment design | Golden sets | LLM Evaluation | LLM-as-judge | Language Models | Large Language Models | Machine Learning | Model Monitoring | Observability | OpenTelemetry | PPV | Prompt engineering | Python | Regression testing | Root Cause Analysis | Root cause | SQL | Sensitivity | Specificity | Statistical significance | Tracing
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Roles
Related jobs
-
Data Scientist USD 128K-214KData Mining | Data Modeling | Data Visualization | Data reporting | Language ProcessingHealth insurance | Holiday pay | Learning and development | Life insurance | Long-term disabilityMid-level Full TimeUSA-MD-Patuxent River22h ago
-
A/B | A/B Testing | B testing | C++ | Canary ReleaseNone Full TimeSeattle, Washington, United States22h ago
-
Research Scientist, Machine Learning USD 117K-173KC++ | Data Analysis | Deep learning | Dimensional data | EvaluationMid-level Full TimeSunnyvale, CA23h ago
-
Computer Vision | Data Curation | Generative Models | Language Models | Large-scaleMid-level Full TimeMenlo Park, CA | Seattle, WA …23h ago
-
Applied Data Scientist and Visualization Specialist - Hardware Engineering Product Design USD 143K-256K3D Data Analysis | 3D data | Anomaly Detection | Computational Geometry | Computer GraphicsMid-level Full TimeSunnyvale1d ago
-
Data Analyst II/Data Scientist II USD 95K-135KAnomaly Detection | Artificial Intelligence | Data Analysis | Data Lifecycle Management | Data analyticsEntry-level Full TimeRosslyn, VA, US R1d ago
-
AI Research Scientist - GenAI USD 165K-185KC plus plus | Deep learning | GenAI | Inference Optimization | Language ModelsMid-level Full TimePittsburgh, PA, United States1d ago
-
AI Research Scientist II USD 139K-223KAction Recognition | Anomaly Detection | Computer Vision | Data labeling | Dataset curationDental insurance | Employee resource groups | Fitness programs | Learning and development programs | Medical insuranceEntry-level Full TimeSeattle, Washington, United States R1d ago
-
Associate, Client Intelligence (Data Scientist) USD 98K-132KAWS | Apache Spark | Azure | Big Data | Cloud platformEmployee resource groups | Matching gift program | Professional development | Sustainability and Climate Change Initiatives | Volunteerism programMid-level Full TimeNew York, NY, US, 100011d ago
-
Data Scientist Staff USD 100K-134KCassandra | Data Modeling | Data Pipelines | Data scrubbing | Hadoop401k company match | Adoption Assistance | Appliance purchase plan | Commuter benefits | Electric vehicle incentive programSenior-level Full TimeFt Worth, Texas, United States1d ago
-
Applied Scientist, AI Economics (TokenOps, FinOps) USD 101K-193KCVaR | Causal Inference | Chance constraints | Conformal Prediction | Data LineageMid-level Full TimeUnited States1d ago
-
Associate Scientist, Computational Biology USD 93K-112KAPIs | AWS | Airflow | Azure | BioinformaticsCompetitive benefits | PTO | Stock Option PlansMid-level Full TimePhiladelphia, PA R1d ago
-
Data Scientist USD 89K-149KC Sharp | Convolutional Neural Network | Dashboards | Data Visualization | Data cleaning401k matching | Dental insurance | Medical insurance | Paid time off | Vision insuranceMid-level Full TimeHouston, TX, United States1d ago
-
Lead Insurance Data Scientist - P&C USD 100K-150KAzure | Dashboarding | Data Analysis | Data Quality | Data Visualization401k matching | Dental insurance | Medical insurance | Mentorship program | Paid parental leaveSenior-level Full TimeRemote US Based R1d ago
-
Staff Data Scientist (Insurance Tech) USD 200K-225KA/B | A/B Testing | B testing | Bayesian statistics | Causal Inference401k matching | Catered lunches | Flexible schedule | Happy hours | Life insuranceSenior-level Full TimeNew York, NY1d ago
-
Senior Manager, Data Science USD 60K-76KAPIs | Analytics engineering | Artificial Intelligence | Cloud Computing | Data Modeling401k | Disability insurance | Employee assistance program | Flexible spending accounts | Health savings accountSenior-level Full TimeUnited States1d ago
-
Staff Data Scientist - Finance & Accounting USD 212K-265KAI | Cycle management | DBT | Data Analysis | Data Modeling16-week parental leave | 401k | Employee assistance program | Equity compensation | Flexible PTOSenior-level Full TimeNew York, NY R1d ago
-
Connected Services Analytics & AI Enablement Lead USD 129K-192KAI Agents | Cloud Computing | Data Engineering | Data Governance | Data QualityDental insurance | Disability insurance | Life insurance | Medical insurance | Paid time offSenior-level Full TimeAuburn Hills, MI, United States1d ago
-
Sr. Data Scientist, Assay Development USD 150K-189KBioinformatics | Confidence Intervals | Control Charts | DMAIC | Design of ExperimentsSenior-level Full TimePalo Alto, CA1d ago
-
ADME | Active Learning | Artificial Intelligence | Biology | CheminformaticsSenior-level Full TimeCambridge, MA1d ago
-
Mid-Data Scientist USD 120K-173KAmazon Web Services | Clustering | Data Quality | Data Visualization | Data Wrangling401k matching | Dental insurance | Health insurance | Life insurance | Paid federal holidaysMid-level Full TimeVienna, VA1d ago
-
Data Science Vice President Branch Analytics USD 135K-230KA/B | A/B Testing | Anomaly Detection | Attribution Modeling | B testingBackup childcare | Financial coaching | Health care coverage | Mental health support | Retirement savings planExecutive-level Full TimePlano, TX, United States1d ago
-
Senior Data Engineer USD 119K-234KArtificial Intelligence | Azure | Data Engineering | Data Science | Data analyticsSenior-level Full TimeUnited States, Washington, Redmond; United States, …1d ago
-
Senior Data Scientist Strategic Data Intelligence USD 92K-152KClassification | Clustering | Data Modeling | Data Normalization | Data VisualizationSenior-level Full Time4400 West 18th Street, Houston, Texas …1d ago
-
Lead Data Scientist II USD 136K-200KExperiment design | Feature Engineering | Machine Learning | Model Governance | Model MonitoringSenior-level Full TimeWilmington, DE1d ago