Generative AI Testing & Prompt Engineer (AI Evaluation, Bias Mitigation & Automation)

Job Summary
Synechron is seeking a proactive and skilled Generative AI Testing & Prompt Engineer to join our innovative technology team. In this role, you will design and execute prompt strategies, develop automation workflows, and evaluate AI outputs to ensure high-quality, unbiased, and reliable generative models. Your expertise will help improve AI performance, safety, and ethical standards, enabling the organization to leverage cutting-edge AI capabilities effectively and responsibly.

Software Requirements

• Required:
  • Python (including libraries such as NumPy, Pandas, and testing frameworks)
  • Bash or shell scripting for automation workflows
  • Version control tools (Git)
  • AI evaluation metrics tools and frameworks for measuring quality, bias, and safety
• Preferred:
  • Experience with AI/ML frameworks such as TensorFlow or PyTorch
  • CI/CD automation tools for seamless testing and deployment workflows
  • Data analysis tools and platforms for metrics development

Overall Responsibilities

• Develop and refine prompts to evaluate the capabilities, accuracy, creativity, and safety of Generative AI models, including NLP and multimodal models.
• Design comprehensive testing strategies across diverse scenarios using prompt variations to assess performance, bias, and ethical considerations.
• Automate testing workflows to streamline data collection, prompt execution, output analysis, and reporting.
• Analyze and interpret AI outputs, providing insights for improvements and bias mitigation.
• Collaborate closely with AI data scientists and developers to tune prompt techniques and enhance model behavior through automation and feedback.
• Document prompt scripts, testing procedures, automation workflows, and test results for transparency and reproducibility.
• Continuously refine prompt engineering and testing methodologies based on latest AI advancements and industry best practices.
• Stay informed about new trends, tools, and ethical considerations related to generative AI, prompt engineering, and automation.

Technical Skills (By Category)

• Programming Languages:
  • Required: Python (advanced scripting and automation capabilities)
  • Preferred: Bash, shell scripting, or other automation languages
• Databases/Data Management:
  • Basic understanding of data storage and querying for managing testing datasets
  • Experience with data labeling and quality control processes
• Cloud Technologies:
  • Preferred: Basic familiarity with cloud platforms (AWS, Azure, GCP) for AI deployment and testing
• Frameworks and Libraries:
  • Required: Generative AI evaluation tools, scripting frameworks for automation, prompt management tools
  • Preferred: TensorFlow, PyTorch, or similar AI/ML libraries
• Development Tools and Methodologies:
  • Required: Version control (Git), scripting for automation, reporting tools
  • Preferred: CI/CD pipelines for automated testing and deployment in cloud or on-prem environments
• Security Protocols:
  • Awareness of data privacy, security standards, and bias mitigation techniques

Experience Requirements

• 3-5 years of professional experience in testing generative AI models, including prompt writing and optimization.
• Proven experience designing prompt strategies for NLP and multimodal models.
• Hands-on experience automating evaluation workflows for AI outputs, bias detection, and safety assessment.
• Familiarity with model evaluation metrics, including relevance, coherence, bias, and safety.
• Industry experience in AI development, testing, or related research preferred; alternative experience with related automation and data analysis roles may be considered.

Day-to-Day Activities

• Develop, test, and optimize user prompts for AI model evaluations across multiple scenarios.
• Automate processes for prompt execution, data collection, and output assessment.
• Monitor and analyze AI model responses for quality, relevance, bias, and ethical compliance.
• Collaborate with data scientists, AI developers, and product teams to implement prompt tuning and automation improvements.
• Document testing strategies, automation scripts, and test outcomes thoroughly.
• Provide actionable feedback to improve model behavior and address bias or safety issues.
• Review industry trends and incorporate new testing tools or methodologies to advance testing capabilities.

Qualifications

• Bachelor’s degree in Computer Science, Data Science, AI, or related field.
• Relevant certifications in AI, data analysis, or automation practices are a plus.
• Demonstrated commitment to continuous learning of AI ethics, bias mitigation, and prompt engineering.

Professional Competencies

• Critical thinking and analytical skills to interpret complex AI outputs and identify issues.
• Problem-solving abilities to develop effective prompts, troubleshoot automation workflows, and improve testing strategies.
• Strong communication skills for documenting processes and collaborating with multidisciplinary teams.
• Ability to adapt rapidly to evolving AI technologies and industry standards.
• Interpersonal skills for stakeholder engagement and feedback provision.
• Effective time management to handle multiple test scenarios and documentation tasks efficiently.

S​YNECHRON’S DIVERSITY & INCLUSION STATEMENT
 

Diversity & Inclusion are fundamental to our culture, and Synechron is proud to be an equal opportunity workplace and is an affirmative action employer. Our Diversity, Equity, and Inclusion (DEI) initiative ‘Same Difference’ is committed to fostering an inclusive culture – promoting equality, diversity and an environment that is respectful to all. We strongly believe that a diverse workforce helps build stronger, successful businesses as a global company. We encourage applicants from across diverse backgrounds, race, ethnicities, religion, age, marital status, gender, sexual orientations, or disabilities to apply. We empower our global workforce by offering flexible workplace arrangements, mentoring, internal mobility, learning and development programs, and more.

All employment decisions at Synechron are based on business needs, job requirements and individual qualifications, without regard to the applicant’s gender, gender identity, sexual orientation, race, ethnicity, disabled or veteran status, or any other characteristic protected by law.

Candidate Application Notice