Member of Technical Staff (Data Scientist, Evals)
Tasks
- Adapt benchmarks into performance measurements
- Architect automated evaluation pipelines
- Build VLM based evaluation solutions
- Create ground-truth datasets
- Define evaluation metrics
- Develop evaluation sets and methods
- Measure impact of tool calls on answer quality
- Review benchmarks and academic evaluations
Perks/Benefits
- N/A
Skills/Tech-stack
AWS | Cloud Data | Cloud Data Stack | Data Science | Data stack | Databricks | Evaluation metrics | Ground Truth | Ground Truth Dataset | LLM-as-a-Judge | Language Models | Large Language Models | Machine Learning | Python | Retrieval | SQL
Education
Regions
Countries
States
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R11d ago
-
Featured Feat. Associate Director, Data Labs USD 167K-167KAWS | Cloud Computing | Compute Infrastructure | Data Analysis | LLM GovernanceConference speaking opportunities | Hybrid work schedule | Media appearancesSenior-level Full TimeWashington, District of Columbia, 20004, United … R24d ago
-
Data Scientist, Product USD 155K-185KArtificial Intelligence | Data Analysis | Data Visualization | Experiment design | Machine LearningMid-level Full TimeMountain View, CA6h ago
-
Sr. Embedded Software/Firmware Engineer USD 180K-240KARM | ASPICE | AUTOSAR | C# | C++Competitive compensation | Dental insurance | Flexible hours | Medical insurance | Pre-IPO stock optionsSenior-level Full TimeUnited States - Remote R7h ago
-
Sr. Software Engineer, AI Infrastructure USD 144K-236KApache Beam | Apache Flink | Apache Spark | C# | C++Career development | Hybrid work | Stock grantsSenior-level Full TimeSunnyvale, CA, United States8h ago
-
Marketing Data Scientist, Measurement & Experimentation USD 117K-193KA/B tests | Attribution calibration | Bayesian Modeling | Bayesian Structural Time Series | Brand LiftMid-level Full TimeNew York, NY, United States8h ago
-
Google Cloud Platform (GCP) Data Engineer USD 100K-203KApache Beam | BigQuery | Broadcast joins | CI/CD | CachingMid-level Full TimeArlington, VA8h ago
-
Google Cloud Platform (GCP) Data Engineer USD 100K-203KApache Beam | BigQuery | CI/CD | Cloud Storage | Cloud platformSecurity clearance providedMid-level Full TimeArlington, VA8h ago
-
AI/ML Engineer USD 91K-184KAutomated Machine Learning | Continuous Deployment | Continuous integration | Data Pipelines | MLOpsMid-level Full TimeArlington, VA9h ago
-
Senior Data Engineer USD 144K-187KAWS Lambda | Amazon DynamoDB | Amazon ECS | Amazon EKS | Amazon S3401k employer contribution | Dental insurance | Flexible Remote Friendly Work Options | Health Savings Account employer contribution | Health insuranceSenior-level Full TimeAustin, Texas11h ago
-
Senior Data Engineer - Finance USD 172K-203KARR | AWS | Apache Iceberg | Apache Spark | CRMDental insurance | Employee discounts | Flexible wellness time off | Health insurance | Hybrid workSenior-level Full TimeHybrid - San Francisco, California R11h ago
-
Senior Data Scientist USD 167K-250KCausal Inference | Econometrics | Experimentation | Forecasting | Machine Learning401k matching | Dental insurance | Employee discounts | Health insurance | Paid time offSenior-level Full TimeUnited States, Los Angeles, CA12h ago
-
Sr. Data Engineer (HYBRID) USD 145K-160KAWS | AWS Glue | AWS Lambda | Amazon Athena | Amazon DynamoDBComplimentary club membership | Hybrid work schedule | Personal training | Pilates | Shop discountsSenior-level Full TimeNew York, NY, United States R12h ago
-
Staff Data Engineer - Data Engineering USD 207K-275KAccess Control | Apache Fluss | Apache Hudi | Apache Iceberg | Apache Paimon401k matching | Dental insurance | Disability insurance | Employee stock purchase program | Flexible PTOSenior-level Full TimeLivingston, NJ / New York, NY …12h ago
-
Cheminformatics | Cloud Computing | Data Analysis | Data Visualization | Deep learningSenior-level Full TimeRedwood City, California, United States12h ago
-
Active Learning | Bioinformatics | Chemical informatics | Cloud Computing | Data integrationSenior-level Full TimeRedwood City, California, United States12h ago
-
Cloud Computing | Data Analysis | Data Visualization | Deep learning | Ensemble MethodsSenior-level Full TimeRedwood City, California, United States12h ago
-
Senior MLOps Engineer I USD 170K-180KAWS | CI/CD | Canary deployments | Data Versioning | Deployment PipelinesAutonomy | Dental insurance | HSA Eligible | Inclusive culture | Medical insuranceSenior-level Full TimeSan Francisco, California, United States12h ago
-
Senior MLOps Engineer I USD 170K-180KAPI Integration | AWS | CI/CD | Canary Deployment | Computer VisionDental insurance | HSA options | Health insurance | Paid training | Stock options (eligibility)Senior-level Full TimeBoston, Massachusetts, United States12h ago
-
AI Developer USD 99K-198KAWS | Agentic AI | Azure | CI/CD | Data LakesHybrid work schedule | Leadership development | Professional developmentMid-level Full TimeWashington, DC12h ago
-
Software Engineer USD 85K-100KBiometrics | Computer Vision | Deep learning | Facial recognition | Image Geolocation401k match | Dental insurance | Disability insurance | Health insurance | Life insuranceMid-level Full TimeSomerville, Massachusetts, United States12h ago
-
AI Engineer I (Contractor) - 40526R0024893 USD 130K-183KAI Agents | AI Inventory | AI monitoring | Documentation | Enterprise SearchHybrid work model | Work standard business hoursEntry-level ContractAustin, TX, United States13h ago
-
Senior-level Full TimeOrlando, FL, United States14h ago
-
AI Requirements Engineer USD 120K-145KAWS | Acceptance criteria | Agile | Analytics | Artificial IntelligenceMid-level Full TimeArlington, VA, United States14h ago
-
Staff Data Scientist, ML (People Analytics & Insights) USD 217K-255KA/B | A/B Testing | Access Control | B testing | BigQuery401k matching | Fertility benefits | Life and disability insurance | Lifestyle spending account | Mental health benefitsSenior-level Full TimeChicago, IL; Menlo Park, CA; New …14h ago