AI Eval / Testing (Eval Engineer)
Tasks
- Automate evaluation and regression testing
- Build AI evaluation pipelines
- Collaborate with cross-functional teams to improve user experience
- Conduct root cause analysis
- Define AI quality metrics and KPIs
- Develop LLM as a judge graders
- Implement CI CD release gates
- Implement observability tracing and session tracking
- Measure factuality faithfulness and toxicity
- Perform adversarial red teaming
Perks/Benefits
- N/A
Skills/Tech-stack
Adversarial Testing | Benchmarking | CI/CD | Git | Grounding | Human-in-the-loop | KPI Monitoring | LLM-as-a-Judge | Latency optimization | Observability | PII Leakage | Prompt engineering | Prompt injection | Pytest | Python | RAG | Red Teaming | Retrieval-Augmented Generation | Robot Framework | Selenium | Session tracking | Software testing | Test automation | The Loop | Tracing
Education
Related jobs
-
.NET | AWS | AWS Lambda | Agentic Workflows | Azure401k | Dental insurance | Medical insurance | PTO | Retirement benefitsSenior-level Full TimeLos Angeles, CA, United States R4h ago
-
Fullstack Engineer, AI Integrations USD 50K-70KAWS | Agile | Alerting | C++ | CSSAgile team environment | Hybrid work | MentorshipEntry-level Full TimeMountain View, CA / San Francisco, … R5h ago
-
Entry-level Full TimeMountain View, CA / San Francisco, … R5h ago
-
Ansible | C# | C++ | CI/CD | EthernetActive Secret clearance requirement | Hybrid work | Travel 5 percent domestic and internationalMid-level Full TimeLexington, MA, United States6h ago
-
API Integration | ARM | Angular | Appian | Azure DevOpsFlexible extensions contract | Hybrid work schedule | Knowledge transfer coaching | Onsite work with mission teamsSenior-level ContractAustin, United States6h ago
-
Senior-level Full TimeUSA-VA-Herndon6h ago
-
Data Science Team Leader USD 165K-165KCI/CD | Cloud platform | Docker | Google BigQuery | Google CloudSenior-level Full TimeDenver, Colorado, United States7h ago
-
Business Operations Manager, Applied AI Engineering USD 160K-185KAI tool integration | Agent Orchestration | Bias Mitigation | Business Strategy | Context engineeringSenior-level Full TimeMenlo Park, CA | Seattle, WA …8h ago
-
Network Engineer, Foundation & Support USD 120K-184KAI Assisted Development | Automation | C# | C++ | Distributed SystemsGlobal team collaboration | Mentorship | On-the-job trainingEntry-level Full TimeDenver, CO | Reston, VA | …8h ago
-
Agentic Workflows | Automated testing | Computer Vision | Data Processing | Function CallingSenior-level Full TimeMountain View, CA, USA8h ago
-
Technical Lead, AI/ML Infrastructure USD 207K-301KC# | C++ | Compute architecture | Cryptography | Distributed SystemsSenior-level Full TimeSunnyvale, CA, USA8h ago
-
Senior-level Full TimeCenter, Center District, IL9h ago
-
Data Engineer USD 96K-137KApache Airflow | Cloud platform | DBT | Git | Google Cloud401k matching | Basic life insurance | Dental insurance | Disability coverage | Medical insuranceMid-level Full TimePiscataway, NJ, US11h ago
-
Principal AI Platform Engineer USD 104K-166KAPIs | Access Control | Audit trails | Data Engineering | Data GovernanceSenior-level Full TimeSan Francisco, CA17h ago
-
Artificial Intelligence Developer (AI) USD 114K-218KAmazon Web Services | C++ | Conda | Data Modeling | ETL401k matching | Employer Covered Dental Insurance | Employer Covered Disability Insurance | Employer Covered Vision Insurance | Employer-covered health insuranceMid-level Full TimeChantilly, VA18h ago
-
Sr. Embedded Software Engineer - Radar & DSP USD 165K-220KAgile | Anomaly Detection | C# | C++ | ClassificationHealth insurance | Onsite work | Professional development | Retirement plansSenior-level Full TimeHuntington Beach, CA18h ago
-
Data Engineer USD 125K-160KAWS | AWS AppFlow | AWS CloudFormation | AWS Glue | AWS LambdaIn-office workSenior-level Full TimeMeridian, ID, US19h ago
-
Data Engineer Senior Principal (Hybrid) USD 144K-195KAmazon S3 | Amazon Web Services | Amazon Web Services (AWS) | Apache Airflow | Apache Flink401k match | Health insurance | Hybrid work | Paid time offSenior-level Full TimeUSA NC Fort Bragg - 2929 … R19h ago
-
Gen AI Engineer USD 112K-168KAKS | AWS | Agile | Agile frameworks | Apache Spark401k match | Dental insurance | Financial education resources | Health insurance | Life insuranceMid-level Full TimeGA-ATLANTA, 740 W PEACHTREE ST NW, …19h ago
-
Lead Cloud Data and AI/ML Engineer, AVP USD 90K-157KAPI | AWS | AWS Lambda | Agentic AI | AirflowDental insurance | Employee assistance program | Family care benefits | Health insurance | Incentive compensationSenior-level Full TimeQuincy, Massachusetts, United States19h ago
-
Machine Learning Engineer USD 137K-275KAWS | C++ | Docker | Java | KubernetesHybrid work | Remote work options | Work-life balanceMid-level Full TimeSeattle (WA), United States19h ago
-
Innovation Senior AI Engineer USD 210K-260KAWS | Agentic Systems | Agno | Apache Airflow | AzureHome office setup budget | Multisport card | Private healthcare | Professional development budgetSenior-level Full TimeBoston, United States19h ago
-
AI Data Scientist USD 75K-102KAPIs | Azure Cognitive | Azure Cognitive Search | Chroma | Chunking401k match | Full-flex work week | Hybrid work | Paid time off | Wellness programsMid-level Full TimeUSA LA Bossier City - 6310 …19h ago
-
Machine Learning Engineer USD 110K-165KAWS | Backend Services | CAD | Computational Geometry | Computer GraphicsCareer advancement | Catered team lunches | Equity ownership | Growth opportunities | Medical/Dental/Vision insuranceMid-level Full TimeSan Mateo, CA19h ago
-
Software Engineer I USD 74K-108KAzure | Cloud platform | Data Preprocessing | Data Structures | Data Structures and Algorithms401k matching | Accident and hospital indemnity | Comprehensive medical, dental, and vision | Corporate fitness program | Flexible time offEntry-level Full TimeDallas, United States19h ago