Natural Language Measurement Specialist
Tasks
- Apply psychometric principles to LLM based feedback and annotation systems
- Build evaluation frameworks linking model behavior to measurement outcomes
- Collaborate with psychometricians data scientists and engineers
- Conduct fairness analyses across subgroups
- Define language based constructs for automated feedback and annotation
- Design and lead validity studies for feedback and annotation
- Design prompt design fine tuning and evaluation for LLM feedback
- Detect and mitigate bias in language based scores
- Develop machine learning models for language measurement
- Document methodologies and research findings
- Establish annotation guidelines and feedback quality criteria
- Integrate feedback and annotation models into scalable systems
- Prepare and curate datasets for measurement studies and model evaluation
- Support reproducible data processing workflows
Perks/Benefits
Skills/Tech-stack
Dimensionality | Experimental Design | Fairness | Fine Tuning | Generalizability theory | Item Response Theory | Language Models | Language Processing | Large Language Models | Machine Learning | Measurement invariance | Natural Language | Natural Language Processing | NumPy | Pandas | Prompt engineering | Psychometrics | PyTorch | Python | Reliability | Response Theory | Statistical Analysis | Validity
Education
Related jobs
-
Featured Feat. Associate Director, Data Labs USD 167K-167KAWS | Cloud Computing | Compute Infrastructure | Data Analysis | LLM GovernanceConference speaking opportunities | Hybrid work schedule | Media appearancesSenior-level Full TimeWashington, District of Columbia, 20004, United … R28d ago
-
Staff Applied Data Scientist, Pricing USD 207K-244KA/B | A/B Testing | B testing | Causal Inference | Data ModelingQuarterly in person surges | Remote-first workSenior-level Full TimeRemote - USA R10h ago
-
Senior Applied AI/ML Scientist - Retailer USD 211K-290KC plus plus | Causal Inference | Deep learning | Experimentation | JavaEquity | Health benefits | Hybrid work | Paid time off | Remote work up to 4 weeks per yearSenior-level Full TimeSan Francisco, CA R12h ago
-
Sr. Data Scientist USD 133K-160KBusiness Intelligence | Data Analysis | Language Models | Large Language Models | Looker401k match | Equity | Flexible time off | Health insuranceSenior-level Full TimeRaleigh, NC R15h ago
-
Mid-level Full TimeRemote, USA R20h ago
-
Generative AI Specialist USD 100K-150KAgent systems | Agentic Workflows | Chunking | Deterministic Evaluation | EmbeddingsSenior-level Full TimeUnited States - Remote R1d ago
-
Large Language Model Specialist USD 100K-150KAdapter-Tuning | Attention Optimization | Benchmarking | DPO | DeepSpeed ZeRODirect W2 position | Remote workSenior-level Full TimeTempe, AZ R1d ago
-
Military Advisor- Data Science/Sentiment Analysis (Part Time/Remote) (Mission Assurance 4)- 29524 USD 106K-150KAPIs | AWS | Azure | BERT | BeautifulSoupEpisodic travel | Remote work | Training opportunitiesSenior-level Part TimeCamp HM Smith, HI, Remote, United … R1d ago
-
Senior Scientist, Data Science - Hybrid BRL 106K-132KAnomaly Detection | Causal Inference | Decision Making | Deep learning | Demand forecasting401k match | Disability insurance | Education assistance | Full health insurance on day one | Incentive planSenior-level Full TimeBoston, MA, US, 02110 R1d ago
-
Principal ML Scientist USD 215K-323KAgent systems | Agentic AI | Claude Code | Cloud Platforms | Data PipelinesEmployee stock purchase plan | Paid time off | Parental leave | Remote work options | Tuition assistanceSenior-level Full TimeUnited States R1d ago
-
Senior Data Scientist - Customer Experience USD 132K-166KA/B | A/B Testing | AI | Airflow | Applied statisticsSenior-level Full TimeUnited States R1d ago
-
Senior Data Scientist - Customer Experience USD 132K-166KA/B | A/B Testing | Airflow | B testing | Causal InferenceBonus program | Equity RSUSenior-level Full TimeUnited States R1d ago
-
Data Scientist II, ML Infrastructure USD 114K-235KApache Airflow | Causal Inference | Code review | Distributed Computing | Feature importanceEntry-level Full TimePalo Alto, CA, US; Remote, US R1d ago
-
Sr. Data Scientist, Elastic USD 175K-200KAWS | Azure | Elasticsearch | Embeddings | Google CloudRemote work | Security clearance required | Travel as neededSenior-level Full TimeWork from home, VA, United States R1d ago
-
Sr. Data Scientist, Elastic USD 180K-200KAWS | Azure | Cloud platform | Data Science | ElasticsearchKnowledge sharing and enablement | Remote work | Travel as neededSenior-level Full TimeWork from home, VA, United States R1d ago
-
Data and Insight Analyst USD 75K-110KAgile | CI/CD | Data Governance | Data Modeling | Data Quality401k | Dental insurance | Disability insurance | Health insurance | Life insuranceMid-level Full TimeRemote, United States R2d ago
-
Lead Data Scientist, Revenue Analytics USD 138K-150KA/B | A/B Testing | B testing | Causal Impact | Data Pipelines401k | Dental insurance | Medical insurance | Paid time off | Stock optionsSenior-level Full TimeRemote, US R2d ago
-
Senior Principal Data Scientist - Remote USD 134K-230KAgentic Systems | Convolutional Neural Networks | Deep learning | Gradient boosting | Language ModelsRemote work flexibilitySenior-level Full TimeSan Diego, California, UNITED STATES; REMOTE R2d ago
-
Senior Lead Applied AI Scientist USD 132K-220KAPIs | Agentic Architectures | Cloud AI | Deep learning | ExperimentationSenior-level Full TimeRemote, US R2d ago
-
Sr. Data Scientist - AI Voice USD 113K-154KCloud platform | Data Pipelines | Deep learning | FHIR | Google Cloud401k match | Dental insurance | Disability insurance | Employee assistance program | Employee discount programSenior-level Full TimeMassachusetts, United States R2d ago
-
Lead Data Scientist - Healthcare USD 150K-174KAWS | Azure | Cloud platform | Deep learning | Demand forecastingCareer development opportunities | Equal employment opportunity | Individual responsibilitySenior-level Full TimeNew Jersey, United States - Remote R2d ago
-
Data Science - Analyst 4 USD 225K-254KA/B | A/B Testing | B testing | Causal Impact | Causal Inference401k eligibility | Medical benefits | Paid time off | Parental leaveMid-level Full TimeSan Jose, United States R2d ago
-
AVP/VP, Underwriting Analytics USD 105K-192KAnomaly Detection | Cloud Computing | Clustering | Data Mining | Data Pipelines401k savings | Employee assistance program | Health and welfare benefits | Hybrid work | Inclusive cultureExecutive-level Full TimeDenver - Lawrence, United States R2d ago
-
Senior Data Scientist USD 160K-180KAzure DevOps | Azure Machine Learning | BigQuery | CI/CD | Cross-validationSenior-level Full TimeUnited States - Remote R2d ago
-
Data Scientist, Core Infrastructure USD 144K-261KCloud infrastructure | Data Modeling | Distributed Computing | Hadoop | Log Analysis401k plan | Company bonus | Equity | Medical, dental, and vision benefits | Wellness stipendsMid-level Full TimeSan Francisco R2d ago