QA AI Testing & Automation Engineer

We are seeking a highly skilled Quality Assurance Engineer with expertise in testing AI models, Large Language Models (LLMs), AI agents, and Generative AI applications. The ideal candidate will have experience in validating AI-driven systems, ensuring model accuracy, performance, fairness, explainability, and robustness. Additionally, the candidate will be responsible for developing automation tests alongside their AI testing responsibilities.
The successful candidate will be self-motivated and proactive, demonstrating a strong sense of ownership and accountability while requiring minimal supervision.

What You Will Do

• Develop and implement QA strategies for AI-powered applications, focusing on accuracy, bias, fairness, robustness, and performance. 
• Design and execute automated and manual test cases to validate AI Agents/LLM models, APIs, and data pipelines and good understanding of data integrity, data models, etc 
• Assess AI models using quality metrics such as precision/recall, and hallucination detection. 
• Test AI models for bias, fairness, explainability (XAI), drift, and adversarial robustness. 
• Validate prompt engineering, fine-tuning techniques, and model-generated responses for accuracy and ethical AI considerations. 
• Conduct scalability, latency, and performance testing for AI-driven applications. 
• Collaborate with data engineers to validate data pipelines, feature engineering processes, and model outputs. 
• Design, develop, and maintain automation scripts using Selenium and Playwright for API and web testing 
• Work closely with cross-functional teams to integrate automation best practices into the development lifecycle. 
• Identify, document, and track bugs while conducting detailed regression testing to ensure product quality. 

What You Will Bring

• Proven expertise in testing AI models, LLMs, and Generative AI applications, with hands-on experience in AI evaluation metrics and testing tools like Arize, MAIHEM, and LangTest. 
• Strong proficiency in Python for writing test scripts and automating model validation, along with a deep understanding of AI bias detection, adversarial testing, model explainability (XAI), and AI robustness. 
• Demonstrate strong SQL expertise for validating data integrity and backend processes, particularly in PostgreSQL and MySQL. 
• Strong analytical and problem-solving skills with keen attention to detail, along with excellent communication and documentation abilities to convey complex testing processes and results.