Test Engineer-AI/LLM
USD 100K-200K Senior-level Full Time
Tasks
- Analyze model failures and edge cases
- Assist prompt variation evaluation
- Benchmark LLM performance
- Develop automated test frameworks
- Document defects test results
- Evaluate LLM performance
- Execute end to end generative AI tests
- Execute scripted LLM evaluations
- Log and analyze failure cases
- Maintain testing logs
- Optimize prompts for target scenarios
- Run performance tests for LLM use cases
- Test LLM accuracy bias safety
- Test LLM integrated APIs
- Test LLM integrated user interfaces
- Validate fine tuned models
Perks/Benefits
Skills/Tech-stack
API Testing | AWS | Agile | Azure | Bias Mitigation | Cloud platform | Data Analysis | Data Pipelines | Data Visualization | GCP | Git | Google Cloud | Google Cloud Platform | Jupyter | LLM Evaluation | Langchain | Learning operations | MLOps | MLflow | Machine Learning | Machine Learning Operations | NoSQL | Postman | Prompt engineering | Pytest | Python | Red Teaming | SQL | Selenium | Statistical Analysis | TruLens | Weights and Biases
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Roles
AI | AI Test Engineer | Engineer | Evaluation Engineer | LLM Evaluation Engineer | QA Engineer | Test Engineer
Regions
Countries
States
Cities
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R28d ago
-
Applied AI Software Engineer III USD 102K-210K.NET | AI Agent | AI agent orchestration | AWS | AWS BedrockSenior-level Full TimeDallas, Texas, United States; Hermitage, Tennessee, …4h ago
-
Data Engineer III USD 110K-218KAnalytics | Dashboard Development | Data Quality | Data integration | Data quality assuranceMentorship | Professional development | Travel opportunitiesSenior-level Full TimeArlington/Rosslyn, Virginia, United States4h ago
-
Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure SQLMid-level ContractRaleigh, United States5h ago
-
Solution Architect/Data Engineer USD 150K-170KAPI Design | API Integration | Apache Kafka | Apache Spark | Artificial Intelligence401k match | Dental insurance | Medical insurance | Paid time off | Professional development and career advancementSenior-level Full TimeUnited States-Illinois-Wood Dale5h ago
-
Solution Architect/Data Engineer USD 150K-170KAPIs | Apache Kafka | Azure | Cloud Computing | Data Modeling401k match | Dental insurance | Medical insurance | Paid time off | Professional development opportunitiesSenior-level Full TimeUnited States-Illinois-Wood Dale5h ago
-
Big Data | C++ | Data Pipelines | Distributed Storage | JavaEntry-level Full TimeSeattle, Washington, United States6h ago
-
Advertising | Big Data | C++ | Data Pipelines | Distributed StorageEntry-level Full TimeSeattle, Washington, United States6h ago
-
Machine Learning Engineer Graduate (E-Commerce Recommendation/Search Alliance) - 2027 Start USD 115K-177KAlgorithms | Big Data | Big Data Engineering | Data Engineering | Data StructuresEntry-level Full TimeSeattle, Washington, United States6h ago
-
Machine Learning Engineer Graduate (E-Commerce Recommendation/Search Alliance) - 2027 Start (PhD) USD 129K-237KAlgorithms | Big Data | Big Data Engineering | Data Engineering | Data StructuresEntry-level Full TimeSeattle, Washington, United States6h ago
-
Machine Learning Engineer Graduate (E-Commerce Recommendation/Search Alliance) - 2027 Start USD 115K-177KAlgorithms | Big Data | Big Data Engineering | Data Engineering | Data StructuresEntry-level Full TimeSan Jose, California, United States6h ago
-
Machine Learning Engineer Graduate (E-Commerce Recommendation/Search Alliance) - 2027 Start (PhD) USD 129K-237KBig Data | Big Data Engineering | Data Engineering | Data Structures | Data structures algorithmsEntry-level Full TimeSan Jose, California, United States6h ago
-
Content Classification | DPO | Data Generation | Data Processing | DeepSpeedDevelopment workshops | Hands-on experience | Industry exposure | Learning programs | Social eventsEntry-level InternshipSan Jose, California, United States6h ago
-
Algorithms | C plus plus | Causal Inference | Data Structures | DebiasingEntry-level Full TimeSan Jose, California, United States6h ago
-
Machine Learning Engineer Graduate (E-Commerce Recommendation Live) - 2027 Start (PhD) USD 129K-246KAdvertising | Algorithms | C++ | Causal Inference | Contrastive LearningEntry-level Full TimeSeattle, Washington, United States6h ago
-
C++ | Causal Inference | Contrastive Learning | Debiasing | Deep learningEntry-level Full TimeSeattle, Washington, United States6h ago
-
Machine Learning Digital Design Engineer USD 142K-203KASIC design | CDC | Clock Domain | Clock Domain Crossing | Computer ArchitectureSenior-level Full TimeSunnyvale, CA7h ago
-
Security Engineer, Applied AI USD 147K-195KAWS | Adversarial research | Adversary simulation | Agent Orchestration | AndroidSenior-level Full TimeMenlo Park, CA | New York, …7h ago
-
Ads ML Developer Experience Tech Lead USD 174K-220KAI Agents | Agent Orchestration | Artificial Intelligence | Bias Mitigation | Build systemsSenior-level Full TimeSunnyvale, CA R7h ago
-
C++ | Compute Allocation | Dashboarding | Data Visualization | Machine LearningMid-level Full TimeMountain View, CA, USA; New York, …7h ago
-
Senior Software Engineer, Generative AI, gUP USD 174K-252KA/B | A/B Testing | B testing | Code review | Computer Vision24x7 support rotationSenior-level Full TimeSunnyvale, CA, USA7h ago
-
Automation | C# | C++ | CSS | Dashboard DevelopmentMid-level Full TimeAnn Arbor, MI, USA7h ago
-
Software Engineer III, BigQuery USD 147K-210KArtificial Intelligence | BigQuery | Cloud Run | Data Storage | Data StructuresSenior-level Full TimeKirkland, WA, USA7h ago
-
Software Engineer, Supply Chain USD 147K-210KC++ | CSS | Cloud platform | Data Analysis | Data StructuresMid-level Full TimeSunnyvale, CA, USA7h ago
-
Software Engineer, Enterprise AI Capabilities USD 174K-252KAgentic AI | Cloud Computing | Distributed Systems | GCP | Generative AIMid-level Full TimeSan Jose, CA, USA7h ago