Test Engineer-AI/LLM
Palo Alto, California, United States
USD 100K-200K Senior-level Full Time
Tasks
- Analyze model failures and edge cases
- Assist prompt variation evaluation
- Benchmark LLM performance
- Develop automated test frameworks
- Document defects test results
- Evaluate LLM performance
- Execute end to end generative AI tests
- Execute scripted LLM evaluations
- Log and analyze failure cases
- Maintain testing logs
- Optimize prompts for target scenarios
- Run performance tests for LLM use cases
- Test LLM accuracy bias safety
- Test LLM integrated APIs
- Test LLM integrated user interfaces
- Validate fine tuned models
Perks/Benefits
Skills/Tech-stack
API Testing | AWS | Agile | Azure | Bias Mitigation | Cloud platform | Data Analysis | Data Pipelines | Data Visualization | GCP | Git | Google Cloud | Google Cloud Platform | Jupyter | LLM Evaluation | Langchain | Learning operations | MLOps | MLflow | Machine Learning | Machine Learning Operations | NoSQL | Postman | Prompt engineering | Pytest | Python | Red Teaming | SQL | Selenium | Statistical Analysis | TruLens | Weights and Biases
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Roles
AI | AI Test Engineer | Engineer | Evaluation Engineer | LLM Evaluation Engineer | QA Engineer | Test Engineer
Regions
Countries
States
Cities
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R28d ago
-
Applied AI Software Engineer III USD 102K-210K.NET | AI Agent | AI agent orchestration | AWS | AWS BedrockSenior-level Full TimeDallas, Texas, United States; Hermitage, Tennessee, …5h ago
-
Data Engineer III USD 110K-218KAnalytics | Dashboard Development | Data Quality | Data integration | Data quality assuranceMentorship | Professional development | Travel opportunitiesSenior-level Full TimeArlington/Rosslyn, Virginia, United States5h ago
-
Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure SQLMid-level ContractRaleigh, United States6h ago
-
AI Enablement and Intelligent Automation Specialist USD 82K-132KAI Governance | Artificial Intelligence | Business Analysis | Copilot Studio | Data SecurityHybrid workMid-level ContractDeerfield Beach, United States R6h ago
-
Solution Architect/Data Engineer USD 150K-170KAPI Design | API Integration | Apache Kafka | Apache Spark | Artificial Intelligence401k match | Dental insurance | Medical insurance | Paid time off | Professional development and career advancementSenior-level Full TimeUnited States-Illinois-Wood Dale6h ago
-
Solution Architect/Data Engineer USD 150K-170KAPIs | Apache Kafka | Azure | Cloud Computing | Data Modeling401k match | Dental insurance | Medical insurance | Paid time off | Professional development opportunitiesSenior-level Full TimeUnited States-Illinois-Wood Dale6h ago
-
Alerting | Canary Deployment | Distributed Systems | High Availability | Machine LearningEntry-level Full TimeSan Jose, California, United States6h ago
-
Asynchronous execution | Benchmarking | CUDA | Cache optimization | Compilation OptimizationEntry-level Full TimeSan Jose, California, United States6h ago
-
C++ | CUDA | CUDA toolkit | Code generation | Computation SchedulingEntry-level Full TimeSan Jose, California, United States6h ago
-
Software Engineer Graduate (Data-Intelligent Creation-AI Platform-Global Vision Engineering) - 2027 Start USD 128K-316KAutomation | CI/CD | Computer Vision | Generative AI | Language ProcessingEntry-level Full TimeSan Jose, California, United States6h ago
-
Applied Machine Learning Production Engineer Graduate (AML-Production Engineer) - 2027 Start USD 128K-256KAutomation | C++ | Distributed Systems | Linux | MonitoringMentorship | Professional developmentEntry-level Full TimeSan Jose, California, United States6h ago
-
Anomaly Detection | Data Pipelines | LLM | Language Processing | Log AnalysisEntry-level InternshipSan Jose, California, United States6h ago
-
Big Data | C++ | Data Pipelines | Distributed Storage | JavaEntry-level Full TimeSeattle, Washington, United States7h ago
-
Advertising | Big Data | C++ | Data Pipelines | Distributed StorageEntry-level Full TimeSeattle, Washington, United States7h ago
-
Machine Learning Engineer Graduate (E-Commerce Recommendation/Search Alliance) - 2027 Start USD 115K-177KAlgorithms | Big Data | Big Data Engineering | Data Engineering | Data StructuresEntry-level Full TimeSeattle, Washington, United States7h ago
-
Machine Learning Engineer Graduate (E-Commerce Recommendation/Search Alliance) - 2027 Start (PhD) USD 129K-237KAlgorithms | Big Data | Big Data Engineering | Data Engineering | Data StructuresEntry-level Full TimeSeattle, Washington, United States7h ago
-
Machine Learning Engineer Graduate (E-Commerce Recommendation/Search Alliance) - 2027 Start USD 115K-177KAlgorithms | Big Data | Big Data Engineering | Data Engineering | Data StructuresEntry-level Full TimeSan Jose, California, United States7h ago
-
Machine Learning Engineer Graduate (E-Commerce Recommendation/Search Alliance) - 2027 Start (PhD) USD 129K-237KBig Data | Big Data Engineering | Data Engineering | Data Structures | Data structures algorithmsEntry-level Full TimeSan Jose, California, United States7h ago
-
Content Classification | DPO | Data Generation | Data Processing | DeepSpeedDevelopment workshops | Hands-on experience | Industry exposure | Learning programs | Social eventsEntry-level InternshipSan Jose, California, United States7h ago
-
Algorithms | C plus plus | Causal Inference | Data Structures | DebiasingEntry-level Full TimeSan Jose, California, United States7h ago
-
Machine Learning Engineer Graduate (E-Commerce Recommendation Live) - 2027 Start (PhD) USD 129K-246KAdvertising | Algorithms | C++ | Causal Inference | Contrastive LearningEntry-level Full TimeSeattle, Washington, United States7h ago
-
C++ | Causal Inference | Contrastive Learning | Debiasing | Deep learningEntry-level Full TimeSeattle, Washington, United States7h ago
-
Machine Learning Digital Design Engineer USD 142K-203KASIC design | CDC | Clock Domain | Clock Domain Crossing | Computer ArchitectureSenior-level Full TimeSunnyvale, CA8h ago
-
Security Engineer, Applied AI USD 147K-195KAWS | Adversarial research | Adversary simulation | Agent Orchestration | AndroidSenior-level Full TimeMenlo Park, CA | New York, …8h ago