AI QA Trainer - LLM Evaluation

Tasks

• Analyze model failures and identify patterns
• Assess safety and reliability
• Automate testing using Python and SQL
• Conduct hallucination detection and factual accuracy testing
• Create evaluation rubrics and benchmarks
• Design evaluation frameworks
• Develop evaluation metrics dashboards and quality monitoring
• Document findings and root cause analysis
• Evaluate large language models
• Execute test plans and regression suites
• Perform adversarial testing and red teaming
• Provide recommendations for prompts and guardrails
• Validate tool usage accuracy and workflow reliability
• Verify grounding and retrieval augmented generation outputs

Perks/Benefits

• Flexible contract
• Remote work

Skills/Tech-stack

AI Safety | Adversarial Testing | Bias Assessment | Experiment tracking | Factual consistency | Hallucination detection | LLM Evaluation | Language Models | Large Language Models | Machine Learning | Prompt engineering | Python | Quality Assurance | RAG | Red Teaming | Retrieval-Augmented Generation | SQL | Software testing | Test automation