Applied Data Scientist, Evaluation & Model Behavior
Tasks
- Define annotation rubrics and guidelines
- Define evaluation metrics for reasoning tool usage and safety
- Design model quality metrics
- Engineer system prompts and few shot examples
- Filter score and select training data
- Investigate benchmark regressions
- Maintain golden reference datasets
- Perform root cause analysis for data prompt and model issues
- Validate metrics against human judgment
- Write Python scripts for data sanitization and pipeline management
Perks/Benefits
- Dental insurance
- Fast Response
- Immigration support
- Medical insurance
- Relocation support
- Vision insurance
Skills/Tech-stack
A/B | A/B Testing | B testing | Data Pipelines | Data Science | Experimental Design | Fine Tuning | Human Feedback | Human-in-the-loop | Language Models | Large Language Models | Learning from Human Feedback | Machine Learning | Model Evaluation | Prompt engineering | Python | Reinforcement Learning | Reinforcement Learning from Human Feedback | Statistical Analysis | Statistics | The Loop
Education
Roles
Regions
Countries
States
Related jobs
-
Automated Machine Learning | Bayesian statistics | Machine Learning | Model Maintenance | Model Validation401k | Dental insurance | Health savings account | Medical insurance | Paid time offSenior-level Full TimeNew York, NY, US, NY 10019 R5h ago
-
Senior Scientist, Multi-Omics Analytics USD 120K-181KATAC-seq | Bioinformatics | Cloud Computing | Cross Study Integration | Data NormalizationHealth insurance | Paid time off | Retirement contributionsSenior-level Full TimeTemecula, California, US, 925905h ago
-
Staff Data Scientist, Ai/Ml Software Engineering USD 154K-231KData Analysis | Data Visualization | Data pipeline | Feature Engineering | Machine LearningRemote work | Travel 10 to 25 percent to customers conferences partner locations and development sitesSenior-level Full Time#, CA, US, # R6h ago
-
Staff, Data Scientist (Ads Analytics) USD 152K-261KAirflow | Cohort Analysis | Data Science | Data Visualization | Experimentation401k plan with company match | Electric Car Charging Station | Employee assistance program | Flexible spending account | Health savings accountSenior-level Full TimeMountain View, USA10h ago
-
Staff, Data Scientist (Ads Analytics) USD 152K-261KA/B | A/B Testing | Airflow | B testing | Cohort Analysis401k match | Commuter benefits | Disability insurance | Electric Car Charging Station | Employee assistance programSenior-level Full TimeMountain View, USA10h ago
-
Data Scientist, AWS Quick Data USD 136K-212KData Pipelines | Data Quality | Generative AI | Human annotation | Language ModelsCareer growth | Flexible work model | Hybrid work | Mentorship | Work-life balanceMid-level Full TimeSanta Clara, California, USA12h ago
-
Data Scientist USD 67K-74KAnalytics | Demand forecasting | Experiment design | Inventory optimization | Latency optimization401k match | Autonomy | Casual work environment | Community outreach | Flexible PTOMid-level Full TimeFranklin, TN or Remote R12h ago
-
Data Analytics, Fund Finance - Assistant Vice President USD 100K-135KAirbyte | DBT | Data Aggregation | Data Governance | Data Lineage100 percent employer paid dental | 100 percent employer paid vision | Employer-Matched Retirement Plan | Flexible remote work Friday | Parental leaveExecutive-level Full TimeSalt Lake City, Utah, United States17h ago
-
Sr Data Analyst/Scientist USD 120K-140KData Visualization | Looker | Machine Learning | Mode | Population HealthDental insurance | Health insurance | Professional development | Remote work | Vision insuranceSenior-level Full TimeRemote-USA R17h ago
-
Senior-level Full TimeCA - San Francisco18h ago
-
Principal Applied Scientist USD 190K-210KAgentic AI | Autogen | CrewAI | Data Quality | Decision support401k match | Employee assistance program | Employee stock purchase plan | Flexible schedule | Flexible spending accountSenior-level Full TimeWork From Home, United States R19h ago
-
Principal Applied Scientist USD 190K-210KAnomaly Detection | Cloud Computing | Data Quality | Data quality assurance | Deep learning401k match | Employee assistance program | Flexible schedule | Medical/Dental/Vision insurance | Paid parental leaveSenior-level Full TimeWork From Home, United States R19h ago
-
Mid-Level Data Scientist USD 106K-179KAPIs | Automation | Data Management | Data analytics | Data integrationMid-level Full TimeSpringfield, VA, United States19h ago
-
Expert Exploitation Specialist/Data Scientist USD 100K-162KAPI Development | Access Control | ArcGIS | ArcGIS Desktop | ArcGIS ServerSenior-level Full TimeARNOLD, MO, United States19h ago
-
Data Scientist USD 110K-176KData Formats | Data Processing | Data Transformation | Database principles | GISHybrid remote onsite workMid-level Full TimeFAIRFAX, VA, United States19h ago
-
Senior Data Scientist USD 130K-196KAnomaly Detection | Automation | Clustering | Data Engineering | Data VisualizationSenior-level Full TimeSpringfield, VA, United States19h ago
-
ADMET | Cheminformatics | Chemoinformatics Databases | Ligand screening | Machine Learning401k | Catered meals | Company events | Dental insurance | Flexible work scheduleMid-level Full TimeNew York20h ago
-
Senior Data Scientist II USD 139K-186KAgentic tools | Airflow | CPT | Claims data | Cloud Computing401k matching | Flexible PTO | Health insurance | Home office stipend | Paid parental leaveSenior-level Full TimeUnited States - Remote R21h ago
-
Artificial Intelligence | C++ | Data Visualization | Data integration | Databases401k match | Medical, dental & vision coverage | PTOSenior-level Full TimeBurke, VA, United States21h ago
-
Staff Data Scientist (Growth & Marketing) USD 190K-270KA/B | A/B Testing | Attribution Modeling | B testing | Causal InferenceSenior-level Full TimeNew York, NY21h ago
-
Staff Data Scientist (Growth & Marketing) USD 190K-270KA/B | A/B Testing | Analytics | Attribution Modeling | B testingSenior-level Full TimeNew York, NY21h ago
-
ATAC sequencing | ATAC-seq | Anndata | Bioconductor | Bulk ATAC sequencingMid-level Full TimeUnited States22h ago
-
Senior Data Scientist USD 148K-263KAWS | Azure | C++ | Cloud platform | ContainerizationDependent Health Benefits | Health insurance | Holiday pay | Learning and development | Life insuranceSenior-level Full TimeUSA-DC-Washington, USA-AZ-Chandler23h ago
-
AWS | Amazon Bedrock | Amazon Textract | BERT | GPTJFully remote | US government clearance supportMid-level Full TimeWashington DC, DC R23h ago
-
Vision Language Models/VLM Research Scientist Graduate (Trust and Safety) - 2026 Start (PhD) USD 137K-237KData Processing | Efficient Inference | Efficient Training | Evaluation | Language ModelsCross-functional collaboration | Research opportunities | Technology incubationEntry-level Full TimeSan Jose, California, United States1d ago