AI Quality Assurance Lead

It's fun to work in a company where people truly BELIEVE in what they're doing!

We're committed to bringing passion and customer focus to the business.

Job Description

AI Quality Assurance Lead (Evaluation & Testing)

This role requires working from our local Hyderabad office 2-3x a week.
Location: Hyderabad, Telangana, India

ABOUT THE TEAM

The Generative AI Quality & Safety team owns ABC Fitness’s evaluation frameworks, testing pipelines, and compliance tooling for AI-driven features. We partner with product, engineering, and legal teams to ensure every LLM interaction meets rigorous standards for accuracy, safety, and performance. As our AI Quality Assurance Lead, you’ll architect hybrid (automated + human) testing systems, define GenAI quality KPIs, and embed Responsible AI principles across our fitness-tech platform.

At ABC Fitness, we love entrepreneurs because we are entrepreneurs. We know how much grit it takes to start your own business and grow it into something that lasts. We roll our sleeves up, we act fast, and we learn together.

WHAT YOU’LL DO

• Design and deploy evaluation pipelines for generative AI systems using tools like OpenAI Evals, Promptfoo, and custom test harnesses.
• Develop hallucination detection workflows and bias-analysis frameworks for LLM outputs across 10+ languages.
• Partner with AI researchers to translate model capabilities into testable requirements for product teams.
• Implement CI/CD-integrated regression testing for AI microservices on AWS/Azure, monitoring model drift and performance degradation.
• Lead bug triage sessions, prioritizing issues impacting user trust, legal compliance, or revenue.
• Document QA protocols, failure modes, and root-cause analyses in our internal knowledge base.

WHAT YOU’LL NEED

• 7+ years in QA/testing roles, with 3+ years focused on AI/ML systems (LLMs, recommendation engines, or conversational AI).
• Hands-on experience with GenAI evaluation tools (LangSmith, Weights & Biases) and statistical analysis (Python, SQL).
• Proficiency in cloud platforms (AWS SageMaker, Azure ML) and containerized testing environments (Docker, Kubernetes).
• Deep understanding of Responsible AI principles—fairness, transparency, privacy—and adversarial testing methodologies.
• Ability to mentor junior engineers and communicate technical risks to non-technical stakeholders.
• Certifications like AWS Certified Machine Learning Specialty or Microsoft AI Engineer are a plus.

WHAT’S IN IT FOR YOU:  

• Purpose led company with a Values focused culture – Best Life, One Team, Growth Mindset 

• Time Off – competitive PTO plans with 15 Earned accrued leave, 12 days Sick leave, and 12 days Casual leave per year 

• 11 Holidays plus 4 Days of Disconnect – once a quarter, we take a collective breather and enjoy a day off together around the globe. #oneteam 

• Group Mediclaim insurance coverage of INR 500,000 for employee + spouse, 2 kids, and parents or parent-in-laws, and including EAP counseling 

• Life Insurance and Personal Accident Insurance 

• Best Life Perk – we are committed to meeting you wherever you are in your fitness journey with a quarterly reimbursement 

• Premium Calm App – enjoy tranquility with a Calm App subscription for you and up to 4 dependents over the age of 16 

• Support for working women with financial aid towards crèche facility, ensuring a safe and nurturing environment for their little ones while they focus on their careers. 

We’re committed to diversity and passion, and encourage you to apply, even if you don’t demonstrate all the listed skillsets! 

ABC’S COMMITMENT TO DIVERSITY, EQUALITY, BELONGING AND INCLUSION: 
ABC is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.  We are intentional about creating an environment where employees, our clients and other stakeholders feel valued and inspired to reach their full potential and make authentic connections. We foster a workplace culture that embraces each person’s diversity, including the extent to which they are similar or different. ABC leaders believe that an equitable and inclusive culture is not only the right thing to do, it is a business imperative. Read more about our commitment to diversity, equality, belonging and inclusion at abcfitness.com  

ABOUT ABC: 

ABC Fitness (abcfitness.com) is the premier provider of software and related services for the fitness industry and has built a reputation for excellence in support for clubs and their members. ABC is the trusted provider to boost performance and create a total fitness experience for over 41 million members of clubs of all sizes whether a multi-location chain, franchise or an independent gym. 

 
Founded in 1981, ABC helps over 31,000 gyms and health clubs globally perform better and more profitably offering a comprehensive SaaS club management solution that enables club operators to achieve optimal performance. ABC Fitness is a Thoma Bravo portfolio company, a private equity firm focused on investing in software and technology companies (thomabravo.com). 

#LI-HYBRID 

If you like wild growth and working with happy, enthusiastic over-achievers, you'll enjoy your career with us!