Research Engineer – Benchmarking, Evals & Failure Analysis
Tasks
- Align evaluation systems with training and product goals
- Analyze data quality and performance trends
- Build evaluation pipelines
- Conduct failure analysis
- Create rubrics and evaluators
- Design benchmarking systems
- Develop scoring dashboards and reporting
- Run LLM evaluations and experiments
Perks/Benefits
Skills/Tech-stack
API Integration | Algorithms | Benchmarking | Cloud infrastructure | Dashboards | Data Structures | Deep learning | Evaluation Pipelines | Experiment tracking | Failure analysis | LLM Evaluation | Language Models | Large Language Models | Machine Learning | NoSQL | Python | SQL | Scoring systems
Education
N/A
Regions
Countries
States
Related jobs
-
CLOUD SOFTWARE ENGINEER - Big Data Cloud - 8+ yrs of Experience - TS/SCI w/Poly clearance is required - TB A USD 213K-219KAngular | Apache Lucene | Apache NiFi | Apache Solr | Aspect Oriented Design401k retirement plan | AD D Insurance | Dental plan | Federal Holidays | Floating holidaysSenior-level Full TimeAnnapolis Junction, United States1h ago
-
Causal Inference | Cross-modal fusion | DPO | Data Modeling | Deep learningEntry-level Full TimeSan Jose, California, United States4h ago
-
Senior Software Engineer, Database Acceleration USD 212K-450KApache Flink | Apache Spark | Big Data | Cloud Native | Data LakeSenior-level Full TimeSan Jose, California, United States4h ago
-
Click Through Rate | Click Through Rate Prediction | Conversion Rate | Conversion Rate Prediction | Data ModelingMid-level Full TimeSan Jose, California, United States4h ago
-
Forward Deployed Engineer, Generative AI, Google Cloud USD 183K-265KAccelerators | C++ | Cloud Architecture | Data Pipelines | Distributed TrainingTravelSenior-level Full TimeAddison, TX, USA; Washington D.C., DC, …5h ago
-
Software Engineer, BigQuery Managed Storage USD 147K-211KAlgorithms | BigQuery | Change Data Capture | Cloud Computing | Data CaptureMid-level Full TimeKirkland, WA, USA5h ago
-
C++ | Data Structures | Data structures algorithms | Debugging | Google CloudMid-level Full TimeSunnyvale, CA, USA5h ago
-
C++ | Data Processing | Data Storage | Debugging | Distributed ComputingSenior-level Full TimeMountain View, CA, USA5h ago
-
Senior Software Engineer, AI/ML USD 174K-252KData Processing | Data Structures | Data Structures and Algorithms | Debugging | Distributed ComputingSenior-level Full TimeMountain View, CA, USA5h ago
-
Forward Deployed Engineer, Applied AI, Google Cloud USD 127K-183KAPI | CRM | Cloud platform | Conversational AI | Conversational agentsHigh-impact role | Opportunity to lead technical delivery | Travel up to 50 percentMid-level Full TimeNew York, NY, USA; Atlanta, GA, …5h ago
-
Software Engineer, AI System Hacker, GenAI, DeepMind USD 174K-252KArtificial Intelligence | C++ | CSS | Data Visualization | Generative AIMid-level Full TimeMountain View, CA, USA5h ago
-
Agent systems | Agentic Workflows | Cloud platform | Cost Per Request | CrewAISenior-level Full TimeNew York, NY, USA; Addison, TX, …5h ago
-
Software Engineer, Applied AI USD 147K-211KAI Agents | Cloud infrastructure | Data Processing | Debugging | Generative AIMid-level Full TimeSunnyvale, CA, USA5h ago
-
Senior Software Engineer, Generative AI USD 174K-252KAgent Development | Agent Development Kit | Agent-based | Agent-based systems | Cloud ComputingSenior-level Full TimeSunnyvale, CA, USA5h ago
-
Data engineer , Machine Learning USD 170K-240KApache Airflow | Apache Spark | Audio Data | Coverage monitoring | DagsterEmployee Assistance Program (EAP) | Employer matching 401k | Health, dental, vision insurance | Unlimited PTOMid-level Full TimeSan Francisco10h ago
-
Data Platform Engineer II USD 100K-113KAzure Data | Azure Data Factory | Azure Key Vault | Azure Managed | Azure Managed IdentitiesDental insurance | Disability insurance | Health insurance | Life insurance | Unlimited paid time offMid-level Full TimeNeedham, MA10h ago
-
AI Analyst USD 101K-165KAI chatbots | API Integration | Agentic AI | Agile | Artificial IntelligenceDental coverage | Disability insurance | Family leave | Flexible spending accounts | Life insuranceMid-level Full TimeKing of Prussia, PA, US, 1940611h ago
-
Senior Data Engineer USD 160K-207KDBT | Data Architecture | Data Governance | Data Observability | Data Quality401k match | Dental insurance | Equity | Family planning resources | Flexible vacation daysSenior-level Full TimeRemote - USA R13h ago
-
Systems Engineer (Network / Storage / Systems) USD 335K-455KAutomation | Bash | Cause analysis | Cluster management | Configuration ManagementHybrid work model | Relocation assistanceSenior-level Full TimeSan Francisco13h ago
-
Machine Learning Engineer USD 138K-183KAWS | Airflow | Amazon Redshift | Amazon SageMaker | Apache Flink401k retirement savings | Health care insurance | Occasional travel for meetings | Paid parental leave | Paid sick timeMid-level Full TimeRemote - US R13h ago
-
Staff Data Engineer USD 160K-207KAnomaly Detection | DBT | Data Governance | Data Modeling | Data Quality401k match | Dental insurance | Family planning resources | Flexible vacation policy | Fully remoteSenior-level Full TimeRemote - USA R14h ago
-
Senior Software Engineer, Data Engineering USD 164K-227KAWS | AWS Glue | Amazon EMR | Amazon Redshift | Apache Airflow401k match | Backup Child Care | Backup elder care | Backup pet care | Chime days paid time offSenior-level Full TimeSan Francisco, CA, USA15h ago
-
Database Engineer USD 162K-203KAWS | Backup and Restore | C++ | Cost Optimization | Data WarehouseCommuter stipend | Generous PTO | Health, dental, vision coverage | Learning and development stipend | Retirement benefitsSenior-level Full TimeSan Francisco, CA; New York, NY16h ago
-
Sr Analyst, Data Engineer USD 76K-143KAPI Integration | AWS | Agile | Batch Processing | CI/CD401k match | Business Casual Attire | Dental insurance | Health insurance | Hybrid work optionSenior-level Full TimeOhio - Columbus, Three Nationwide Plaza, … R16h ago
-
Data Engineer (Denver, Colorado, USA) USD 120K-150KApache Spark | Data Engineering | Data Governance | Data Pipelines | Data Quality401k retirement plan | Client-facing work | Health insurance | Paid vacationSenior-level Full TimeDenver, Colorado, United States16h ago