Research Scientist, LLM Evaluation & Post-Training
Tasks
- Analyze model behavior and failure patterns
- Build benchmark datasets and test suites
- Conduct human evaluation and rubric design
- Create technical documentation and research reports
- Define LLM evaluation research agenda
- Design LLM post training experiments
- Design scoring methods and evaluation metrics
- Develop LLM evaluation frameworks
- Develop scalable evaluation and post training pipelines
- Engage with customer technical stakeholders on evaluation goals
- Integrate human in the loop and synthetic evaluation strategies
- Provide actionable model improvement recommendations
- Publish and present research findings
- Run robustness and stress testing
Perks/Benefits
- N/A
Skills/Tech-stack
AI Feedback | Alignment | Benchmarking | Context evaluation | Deep learning | Direct Preference Optimization | Error Analysis | Experimental Design | Fine Tuning | Generalized Reward Policy Optimization | Hugging Face | Human Feedback | Human evaluation | Inter-rater reliability | JAX | LLM Evaluation | Learning from Human Feedback | Long Context | Long Context Evaluation | Machine Learning | Multimodal evaluation | Policy Optimization | Preference optimization | Proximal Policy Optimization | PyTorch | Python | RAG | Reinforcement Learning | Reinforcement Learning from AI Feedback | Reinforcement Learning from Human Feedback | Robustness Testing | Safety | Significance Testing | Statistical Analysis | Supervised Fine Tuning | TensorFlow | Uncertainty Quantification | Vector Databases
Education
Related jobs
-
Applied Scientist USD 126K-206KAI Agents | Causal Inference | DBT | Diff-in-diff | Documentation401k matching | Cell phone subsidy | Commuter benefits | Dental insurance | Disability insuranceSenior-level Full TimeRemote - United States R12h ago
-
Data Scientist USD 100K-155KAWS SageMaker | Airflow | Amazon Redshift | DBT | EmbeddingsRemote work opportunity | Travel up to quarterlySenior-level Full TimeRemote (North America) R17h ago
-
Transportation Data Scientist USD 116K-120KData Analysis | Data Management | Data Visualization | Machine Learning | PythonHybrid schedule | PTO | Paid Background Check | Paid Holidays | Transit benefitsMid-level Full TimeDenver, CO, US R1d ago
-
Data Scientist I - Card Fraud Analytics - Atlanta, GA USD 103K-140KCalculus | Data Cleansing | Data Preparation | Data Visualization | Econometrics401k plan | Dental insurance | Disability insurance | Life insurance | Medical insuranceSenior-level Full TimeAtlanta GA - Telecommuter, United States R1d ago
-
Principal Applied Scientist USD 206K-388KEntity Grounding | Entity Linking | GNNs | Graph Embeddings | Graph Neural NetworksSenior-level Full TimeSan Jose, United States R1d ago
-
Data Scientist, Marketing USD 140K-348KAttribution | Causal Inference | Experimentation | Machine Learning | Marketing AnalyticsCell phone reimbursement | Generous PTO | Health/Dental/Vision | Learning and development stipend | Mental health and wellness benefitsMid-level Full TimeSan Francisco, CA • New York, … R1d ago
-
Marketing Data Scientist USD 145K-196KA/B | A/B Testing | Attribution | B testing | Causal InferenceSenior-level Full TimeRemote, US R1d ago
-
Senior Data Scientist, Risk USD 163K-225KCompliance Risk | Credit Risk | Fraud Detection | Python | Risk Management401k | Dental benefits | Generous vacation | Health benefits | Paid parental leaveSenior-level Full TimeSan Francisco, New York or Remote … R1d ago
-
Senior Data Scientist, Pricing USD 185K-225KA/B | A/B Testing | B testing | Causal Inference | Cohort Analysis401k matching | Annual company offsite event | Commuter benefits | Hybrid work | Lunch and snacksSenior-level Full TimePalo Alto, CA R1d ago
-
Senior Data Scientist, Product USD 185K-225KA/B | A/B Testing | B testing | Causal Inference | Cohort Analysis401-K matching | Career growth support | Commuter benefits | Company offsite event | Health benefitsSenior-level Full TimePalo Alto, CA R1d ago
-
Senior Data Scientist, Risk USD 185K-225KA/B | A/B Testing | A/B testing frameworks | B testing | Causal Inference401k matching | Annual company offsite event | Commuter benefits | Health benefits options | Hybrid workSenior-level Full TimePalo Alto, CA R1d ago
-
AI Research Scientist, Applied AI USD 120K-170KAPI Integration | Agent systems | Bayesian Methods | CI/CD | Code ReviewsEquity compensation | Remote work | Technical publications supportEntry-level Full TimeRemote, United States R1d ago
-
Senior Consultant Data Scientist USD 110K-164KAWS | Agile | Apache Spark | CI/CD | Data QualityBonus eligibility | Care Com Membership | Discounted pet insurance | Emergency child or elder care days | Employee wellness eventsSenior-level Full TimeArlington, Virginia, United States, Remote Flexible R1d ago
-
Action Recognition | Anomaly Detection | Computer Vision | Deep learning | Edge ComputingDental insurance | Emotional & mental wellness support | Fitness programs | Learning and development | Medical insuranceSenior-level Full TimeSeattle, Washington, United States R1d ago
-
Forward Deployed AI Engineer USD 175K-220KAI Risk | AI Risk Metrics | Agentic AI | Causal Inference | Data PipelinesFlexible paid time off | Health and wellness stipend | Home office stipend | Learning stipend | MacBook Pro providedMid-level Full TimeUnited States R1d ago
-
Staff Applied Scientist USD 191K-287KApproximate Dynamic Programming | Chain management | Decision analysis | Dynamic Programming | Forecasting401k matching | Dental insurance | Flexible paid time off | Health insurance | Home office stipendSenior-level Full TimeRemote - US R1d ago
-
Data Scientist USD 155K-170KApache Airflow | DBT | Experiment tracking | Feature Engineering | GitHubFlexible remote daysSenior-level Full TimeGlendale, CA, United States R1d ago
-
Data Modeling, Fraud Analytics Consulting (Remote) USD 104K-174KAWS | Data Modeling | Data Science | Fraud Detection | Machine LearningDental insurance | Flexible time off | Flexible work environment | Hybrid work | Medical insuranceMid-level Full TimeUnited States, UNITED STATES, United States R1d ago
-
Senior Data Scientist USD 145K-170KDecision analysis | Ensemble learning | Forecasting | Integer Programming | Linear ProgrammingHealth care | Paid time off | Retirement | Sick leave | TelecommutingSenior-level Full TimeSt. Louis, Missouri, US R2d ago
-
Data Scientist - Materials R&D - Remote-Travel USD 110K-150KAWS | Azure | Classification | Cloud Computing | ClusteringCareer development | Family support benefits | Health benefits | Remote work options | Training opportunitiesSenior-level Full TimeMarysville, MI, 48040, US R2d ago
-
Senior Data Scientist USD 155K-195KAWS SageMaker | AWS Step Functions | Apache Airflow | Batch inference | CI/CDSenior-level Full TimeUSA - Remote, United States R2d ago
-
Artificial Intelligence | Cloud Computing | Data Pipelines | Data Processing | Data analyticsEmployee wellness initiative | Insurance benefits | Paid Holidays | Paid sick leave | Paid vacationSenior-level Full TimeCorpus Christi, TX, United States R2d ago
-
Lead Data Scientist, Stars Analytics USD 142K-195KData Analysis | Data Modeling | Data reporting | Databricks | Machine LearningSenior-level Full TimeRemote US, United States R2d ago
-
Associate Data Scientist USD 94K-175KAnomaly Detection | Artificial Intelligence | Classification | Community Detection | Computer VisionFlexible vacation | Flexible work arrangements | Headspace app access | Mental health days | Paid volunteer daysMid-level Full TimeUnited States of America, McLean, Virginia R2d ago
-
Artificial Intelligence | Cloud Computing | Data Processing | Data analytics | Geospatial DataDental insurance | Employee wellness initiative | Health insurance | Life insurance | Long-term disability insuranceSenior-level Full TimeCorpus Christi, TX, United States R2d ago