Natural Language Measurement Specialist
Tasks
- Apply psychometric principles to LLM based feedback and annotation systems
- Build evaluation frameworks linking model behavior to measurement outcomes
- Collaborate with psychometricians data scientists and engineers
- Conduct fairness analyses across subgroups
- Define language based constructs for automated feedback and annotation
- Design and lead validity studies for feedback and annotation
- Design prompt design fine tuning and evaluation for LLM feedback
- Detect and mitigate bias in language based scores
- Develop machine learning models for language measurement
- Document methodologies and research findings
- Establish annotation guidelines and feedback quality criteria
- Integrate feedback and annotation models into scalable systems
- Prepare and curate datasets for measurement studies and model evaluation
- Support reproducible data processing workflows
Perks/Benefits
Skills/Tech-stack
Dimensionality | Experimental Design | Fairness | Fine Tuning | Generalizability theory | Item Response Theory | Language Models | Language Processing | Large Language Models | Machine Learning | Measurement invariance | Natural Language | Natural Language Processing | NumPy | Pandas | Prompt engineering | Psychometrics | PyTorch | Python | Reliability | Response Theory | Statistical Analysis | Validity
Education
Related jobs
-
Senior Research Engineer / Scientist - Storage for LLM USD 207K-368KCUDA | Caching | Data Structures | Distributed Systems | Eviction PolicyConference attendance | Open source contributions | Research resourcesSenior-level Full TimeSeattle, Washington, United States R8h ago
-
Data Scientist, Product Analytics USD 173K-204KAgent Orchestration | Bias Mitigation | Data Mining | Experimentation | Machine LearningCareer growthMid-level Full TimeBellevue, WA | Menlo Park, CA … R9h ago
-
Senior Data Scientist, Engine Systems USD 221K-263KA/B | A/B Testing | B testing | Causal Inference | ConcurrencySenior-level Full TimeSan Mateo, CA, United States R13h ago
-
Data Scientist USD 140K-165KA/B | A/B Testing | Amazon Redshift | Apache Spark | B testingFlexible working hours | Product-driven culture | Vacation policyMid-level Full TimeRemote US R14h ago
-
Senior Data Scientist, Growth USD 200K-260KA/B | A/B Testing | B testing | Causal Inference | DBT401k plan | Dental insurance | Education stipend | Healthy lunches | Home office improvement stipendSenior-level Full TimeSan Francisco, CA R19h ago
-
Senior Data Scientist, Growth USD 200K-260KA/B | A/B Testing | B testing | Behavioral analytics | Causal Inference401k matching | Education stipend | Healthy lunches | Home office improvement stipend | Hybrid workSenior-level Full TimeMountain View, CA R19h ago
-
Mid-level Full TimeSan Jose, United States R21h ago
-
Applied AI Specialist USD 171K-200KAPIs | Anomaly Detection | Cloud Native | Cloud Native Architecture | ComplianceCareer development | Employee resource groups | Flexible work from home | Generous PTO | Internet reimbursementSenior-level Full TimeUS-Washington-Remote, United States R21h ago
-
Principal Applied Scientist, Agentic AI USD 191K-305KAblation Studies | Agent architecture | Agentic AI | Artificial Intelligence | BenchmarkingSenior-level Full TimeRemote-USA, United States R21h ago
-
Strategy Data Scientist - Compliance USD 95K-114KBusiness Intelligence | Cause analysis | DBT | Data Modeling | Data Pipelines401k retirement plan | Dental insurance | Disability insurance | Employee assistance program | Employee recognition programMid-level Full TimeUnited States R21h ago
-
Data Scientist USD 140K-180KArcGIS API | ArcGIS API for Python | ArcGIS Enterprise | ArcGIS Maps | ArcGIS Maps SDK401k retirement plan | Accident Hospital | Critical illness | Employee assistance program | Health spending accountSenior-level ContractJuno Beach, Florida, USA R23h ago
-
Staff AI Scientist USD 170K-225KAdversarial Training | BERT | Data Generation | Data labeling | Dataset development401k plan | Annual events | CalTrain pass | Fastrak reimbursement | Fitness reimbursementSenior-level Full TimePalo Alto R1d ago
-
Staff AI Scientist USD 199K-228KCausal Inference | Collaborative Filtering | Constrained decoding | Counterfactual evaluation | EmbeddingsDental insurance | Employee Health Ring | Employee discounts | Flexible wellness time off | Health insuranceSenior-level Full TimeRemote - United States R1d ago
-
Staff Data Scientist, EGRI USD 144K-236KA/B | A/B Testing | B testing | Data Storytelling | Data VisualizationHealth and wellness programs | Time away from workSenior-level Full TimeSunnyvale, CA, United States R1d ago
-
Data Scientist USD 90K-120KAWS | Agentic AI | Azure | Cypress | Data Analysis401k employer contribution | Remote work optionMid-level Full TimeRemote, USA R1d ago
-
Applied Scientist II USD 167K-220KAgentic AI | Amazon SageMaker | Computer Vision | Data Science | Deep learningHybrid work model | Mentorship programMid-level Full TimeSeattle, Washington, United States R1d ago
-
Associate Data Scientist USD 56K-76KData Modeling | Forecasting | Machine Learning | Optimization | Price elasticity401k | Certification reimbursement | Cruise and travel privileges | Employee stock purchase plan | Health benefitsMid-level Full TimeSeattle, WA, United States R1d ago
-
Senior Applied Scientist USD 144K-236KCausal Inference | Data Analysis | Data Modeling | Experimentation | JavaSenior-level Full TimeMountain View, CA, United States R1d ago
-
Senior Data Scientist - Customer Experience USD 132K-166KA/B | A/B Testing | Airflow | Applied statistics | B testingSenior-level Full TimeUnited States R1d ago
-
IL0219 – Data Scientist. USD 128K-128KAWS Lambda | AWS Step Functions | Agile methodology | Amazon Athena | Amazon QuickSightMid-level Full TimeWarrenville, IL, United States R1d ago
-
Mid-level Full TimeTel Aviv-Yafo, Tel Aviv District, IL R1d ago
-
Senior Associate, Media Optimization USD 45K-90KAd Server | Amazon Ads | BigQuery | DOOH | DSPDental insurance | Employee mental health awareness days | Employee resource groups | Hybrid work | Medical insuranceSenior-level Full TimeNew York, United States R1d ago
-
Senior Data Scientist - Consumer USD 190K-267KA/B | A/B Testing | B testing | Causal Inference | Dashboards401k employer match | Caregiving support | Comprehensive healthcare | Family planning support | Flexible vacationSenior-level Full TimeRemote - United States R1d ago
-
Senior Data Scientist, Ads USD 190K-267KAnomaly Detection | BigQuery | Causal Inference | Experimental Design | Hadoop401k employer match | Caregiving support | Family planning support | Flexible vacation | Gender-affirming careSenior-level Full TimeRemote - United States R1d ago
-
Data Scientist USD 160K-245KAmazon EFS | Amazon S3 | BI Dashboards | Data Modeling | Data Pipelines401k matching | Hybrid work model | In-office meals | Paid parental leave | Pension contributionsSenior-level Full TimeNew York, New York, United States R1d ago