Enterprise Generative AI Testing and Evaluation

Background: 

Mercy Corps is a leading global organization powered by the belief that a better world is possible. In disaster, in hardship, in more than 40 countries around the world, we partner to put bold solutions into action — helping people triumph over adversity and build stronger communities from within. 

The Technology for Development (T4D) team is housed within the Office of the CFAO alongside the Global Information Technology department. T4D collaborates with Mercy Corps’ teams and external partners to unlock new possibilities to improve program quality, impact, and innovation, helping improve more lives through the power of technology and data. Our mission and purpose are to drive sustainable program impact through responsible application of existing and emerging digital technologies. The IT division ensures that Mercy Corps’ global workforce has the infrastructure, security, support, and technology they need to fulfill their critical missions. Both IT and T4D are working together to identify, test, and incubate key implementations of enterprise AI at Mercy Corps. 

Purpose / Project Description: 

This consultancy will support IT by designing and implementing a user-testing protocol based on Mercy Corps’ current framework for Generative AI. The consultant will evaluate tools for the implementation and use of Generative AI at the enterprise level. The consultancy will focus on determining enterprise-level use cases at Mercy Corps for Generative AI, evaluate the utility of an existing custom-built platform (Digital Library Chatbot) against these use cases, and develop/execute a comparative testing format for at least two commercially available Generative AI solutions.

Consultant Activities: 

•  Leveraging previous work on Generative AI testing frameworks, define a clear testing process for enterprise AI tools

•  Conduct guided and/or independent testing on the Digital Library Chatbot

•  Recommend a procurement strategy for testing at least two commercially available enterprise Generative AI solutions (e.g. CoPilot, ChatGPT, Amazon Q)

•  Conduct guided and/or independent testing on enterprise Generative AI Solutions

•  Conduct a comparative analysis of enterprise solutions including financial comparisons

Consultant Deliverables: 

The Consultant will provide as final deliverables: 

•  Report detailing the following elements: 

•  Results of user testing for the Digital Library Chatbot and concise recommendations on enterprise-wide rollout 

•  Results of user testing for commercially available enterprise Generative AI solutions and recommendations for procuring an enterprise-wide Generative AI solution 

Timeframe / Schedule:  

Consultant will work with the IT Director of Data Systems and T4D Director of Data Science to develop a timetable and workplan for the deliverables in the first week of the consultancy during the kickoff meeting. 

The projected start date for this consultancy is early May and it will conclude June 30, 2025

LOE is approximately 30-45 days within a 60-day period, with flexibility depending on daily rate.

Working hours are flexible and we love a sync communication, but this position will require at least some overlap with US Mountain time zone during the week 

The Consultant will report to:

•  Director of Data Science, T4D 

The Consultant will work closely with: 

•  IT, T4D, other Mercy Corps field and HQ elements

Required Experience & Qualifications: 

Required 

•  Bachelor's degree or higher in a relevant field; direct subject matter experience will be considered in lieu of educational attainment

•  Strong experience and demonstrated skills in technology evaluation especially at the enterprise level, participatory research, and the use of Generative AI for aid/development

•  3-5 years of relevant experience with at least one year of specific experience related to IT at aid/development organizations and/or Technology for Development

Desired 

•  Strong preference given to those with experience and familiarity with Mercy Corps

•  Experience with AI and its potential role in international development

•  Familiarity or experience with commercial off the shelf solutions for Generative AI

•  Familiarity or experience with software development and ICT at the enterprise level

Documents Comprising the Proposal 

Please submit the following documentation for the proposal:

•  CV & Cover Letter 

•  Day Rate (please include your application or your submission will not be considered)

Diversity, Equity & Inclusion 

Achieving our mission begins with how we build our team and work together. Through our commitment to enriching our organization with people of different origins, beliefs, backgrounds, and ways of thinking, we are better able to leverage the collective power of our teams and solve the world’s most complex challenges. We strive for a culture of trust and respect, where everyone contributes their perspectives and authentic selves, reaches their potential as individuals and teams, and collaborates to do the best work of their lives.

We recognize that diversity and inclusion is a journey, and we are committed to learning, listening and evolving to become more diverse, equitable and inclusive than we are today.

Equal Employment Opportunity 

We are committed to providing an environment of respect and psychological safety where equal employment opportunities are available to all. We do not engage in or tolerate discrimination on the basis of race, color, gender identity, gender expression, religion, age, sexual orientation, national or ethnic origin, disability (including HIV/AIDS status), marital status, military veteran status or any other protected group in the locations where we work.

Safeguarding & Ethics 

Mercy Corps team members are expected to support all efforts toward accountability, specifically to our stakeholders and to international standards guiding international relief and development work, while actively engaging communities as equal partners in the design, monitoring and evaluation of our field projects. Team members are expected to conduct themselves in a professional manner and respect local laws, customs and MC's policies, procedures, and values at all times and in all in-country venues.