Senior Data Scientist
Tasks
- Analyze trace data with large scale tooling
- Author binary first rubrics
- Build LLM as judge grader prompts
- Conduct A B testing and statistical inference
- Curate evaluation datasets from customer and production signals
- Develop evaluation frameworks for LLM systems
- Ensure dataset reproducibility and trace hygiene
- Generate synthetic evaluation data
- Orchestrate human labeling and calibration sprints
- Partner with PMs to co develop evaluation scenarios
- Prevent reward hacking and calibration drift
- Track grader performance with human match rate and TPR TNR
Perks/Benefits
- N/A
Skills/Tech-stack
A B Testing Metrics | A/B | A/B Testing | B testing | Data Pipelines | Data Quality | Few-Shot Learning | Few-shot | Human-in-the-loop | LLM Evaluation | Language Models | Large Language Models | Machine Learning | Metadata Design | PII compliance | Prompt engineering | Python | Retrieval metrics | SQL | Statistical inference | Synthetic data | TNR | TPR | Testing metrics
Education
Roles
Related jobs
-
Junior Data Scientist USD 87K-101KA/B | A/B Testing | B testing | Classification | Data Analysis401k employer match | Childcare reimbursement | Company events | Company-paid parking | Comprehensive health coverageEntry-level Full TimeSan Diego, CA, United States10h ago
-
Senior Data Scientist USD 200K-240KAWS | Heuristics | Machine Learning | Optimization | Pandas401k match | Flexible PTO | Flexible spending accounts | Medical, dental, vision plans | Teladoc HealthSenior-level Full TimeNew York City, New York14h ago
-
Senior Staff Marketing Data Scientist, AI Enablement USD 185K-212KAgent Based Workflows | Agent-based | Automation | Causal Inference | ComplianceSenior-level Full TimeUnited States16h ago
-
Senior Computational Biologist – Target ID USD 151K-230KAgentic Workflows | Causal Inference | Cloud Computing | Data integration | EHRBonus | Comprehensive benefits package | Equity compensationSenior-level Full TimeNew York, New York; Salt Lake …16h ago
-
Senior Staff Engineer - Data Scientist USD 150K-190KAmazon Web Services | Anomaly Detection | Apache Airflow | Apache Kafka | Apache SparkContract-to-hire | Remote work | Travel opportunitiesSenior-level Full TimeRemote, REMOTE, United States R16h ago
-
AWS | Big Data | ETL | Git | GitHub401k match | Collaborative team culture | Flexible remote work environment | Healthcare benefits | Professional developmentSenior-level Full TimeColorado R18h ago
-
Business Analytics Intern – Sales & CX USD 40K-50KDashboarding | Data Cleansing | Data Preparation | Data Processing | Data TransformationEntry-level Full Time InternshipCarlisle, PA, United States20h ago
-
Senior Data Scientist USD 156K-224KApache Spark | Backtesting | Bayesian Methods | BigQuery | Data ProcessingSenior-level Full TimeUS-CA-Menlo Park21h ago
-
Data Scientist - TikTok Integrity and Safety (San Jose) USD 150K-240KA/B | A/B Testing | Anomaly Detection | B testing | Causal InferenceMid-level Full TimeSan Jose, California, United States22h ago
-
AI Research Scientist, Audio-Visual Understanding, FAIR USD 147K-208KAudio Visual | Audio-Visual Learning | Computer Vision | Data Analysis | Deep learningMid-level Full TimeMenlo Park, CA | New York, …22h ago
-
Research Scientist USD 150K-251KAgent Orchestration | Automatic Speech Recognition | Conversational AI | Language Models | Language ProcessingMid-level Full TimeMenlo Park, CA22h ago
-
AI Research Scientist, Robotics USD 300K-400KComputer Vision | Deep learning | Human robot | Human-Robot Interaction | Imitation LearningExecutive-level Full TimeRedmond, WA | Burlingame, CA22h ago
-
AI Research Scientist, MRS AI (PhD) USD 117K-173KData Analysis | Deep learning | Dimensional data | Experiment design | High Dimensional DataEntry-level Full TimeMenlo Park, CA22h ago
-
Machine Learning Hardware Research Scientist USD 111K-145KASIC design | Architecture Search | C# | C++ | CPUEntry-level Full TimeSunnyvale, CA22h ago
-
AI Research Scientist, Media Data Research - MSL FAIR USD 147K-208KComputer Vision | Data Curation | Data Generation | Data Scaling Laws | Data scalingEntry-level Full TimeMenlo Park, CA22h ago
-
Computer Science Research - US - IC5 USD 166K-244KData Pipelines | Deep learning | Entity recognition | Generative Models | Image-to-videoKnowledge sharing | Mentoring | Open source contributionsMid-level Full TimeBellevue, WA | Menlo Park, CA22h ago
-
DPO | Data Generation | Data alignment | Deep learning | Human FeedbackSenior-level Full TimeMenlo Park, CA22h ago
-
Data Scientist, Product, Google Voice USD 192K-278KData Analysis | Data Visualization | ETL | Machine Learning | PythonSenior-level Full TimeRaleigh, NC, USA; Durham, NC, USA23h ago
-
Search Ads Data Scientist, Product, Gmail Ads USD 138K-198KCausal Inference | Data Analysis | Experiment design | Metrics | PythonMid-level Full TimeMountain View, CA, USA23h ago
-
Senior Data Scientist - AI/ML Engineer USD 185K-250KAPI Development | AWS | Adapter Training | Agentic AI | Cloud NativeAccess to AI resources | Conference Presenting | Conference attendance | Learning budget | Team offsiteSenior-level Full TimeSunnyvale Office1d ago
-
Computer Vision Scientist (multiple levels/openings) USD 165K-219KC++ | Computer Vision | Deep learning | Machine Learning | Neural Networks401k matching | Commuter benefits | Employee referral program | Flexible spending accounts | Life insuranceSenior-level Full TimeSanta Clara, CA1d ago
-
Sr Data Scientist USD 135K-165KAzure | Azure DevOps | Caret | Databricks | Delta LakeAgile environment | Cross-functional collaboration | Technical documentation in wikis | Version controlled repositoriesSenior-level Full TimeDes Moines, IA, United States1d ago
-
Data Scientist - multiple level USD 56K-75KCloud Computing | Clustering | Data Analysis | Data Engineering | Exploratory Data Analysis401k matching | Dental insurance | Life insurance | Medical insurance | Paid HolidaysEntry-level Full TimeMason, MI, US1d ago
-
Data Scientist USD 131K-180KAzure Machine Learning | Data Wrangling | Data cleaning | Feature Engineering | Machine LearningRemote workSenior-level Full TimeUnited States - Remote R1d ago
-
Mid-level Full TimeUSA VA Virginia Beach - 1300 …1d ago