AI Evaluation Manager

About the role

The GSMA's Mobile for Development (M4D) team is hiring an AI Evaluation Manager in Nairobi for a short-term contract until the end of March 2027, with possible extension. The role sits at the intersection of AI and impact measurement: it helps the Innovation Fund understand how grantees' AI solutions work, whether they perform as intended and how they contribute to impact. It covers model evaluation, product evaluation, MEL planning and thought leadership on AI for impact.

Key responsibilities

• Evaluate grantees' AI solutions by reviewing their architecture (foundation models, prompting, fine-tuning, RAG, agents, guardrails) and assessing model and product performance.
• Design and run Level 1 model evaluations, including evaluation datasets, metrics such as accuracy, relevance, safety and hallucination rates, and LLM-as-judge or human review.
• Support product evaluations, defining usage, engagement and retention metrics and helping grantees use A/B testing and product analytics.
• Advise grantees on practical AI evaluation approaches and build their capacity, drawing on the CGD / Agency Fund Generative AI Evaluation Playbook.
• Draft MEL plans and theories of change for each grantee, manage grantee KPI reporting and develop baseline and endline survey plans with external suppliers.
• Analyse portfolio data, build the Fund's evidence base and produce reports, blogs and briefs on AI for impact.
• Represent Mobile for Development externally at events, webinars and panels, and build relationships with funders, researchers and the wider AI-for-development community.

Required profile

• Solid understanding of the AI stack and comfort getting under the hood of a grantee's AI solution, without needing to be a machine learning engineer.
• Ability to evaluate an AI model and product confidently and explain findings to technical and non-technical audiences.
• Experience with start-ups, SMEs or tech teams, ideally in LMICs across Africa and Asia, in evaluation, data, product or MEL roles.
• Track record of producing thought leadership and speaking publicly; experience building stakeholder relationships.
• Desired: survey design and primary data collection (sample sizes above 100), training or capacity-building experience.

Required skills

• Generative AI and LLMs: foundation models, prompting, RAG, fine-tuning, agents and guardrails.
• Hands-on AI evaluation: evaluation datasets, performance and safety metrics, LLM-as-judge, usage and engagement analysis.
• Social impact measurement: theories of change, MEL frameworks, survey and experimental evaluation.
• Proficiency in Excel; Python, R or SQL, or AI evaluation and observability tools, are a strong plus.

What we offer

• Access to the mobile industry and global, industry-changing projects.
• Architect-designed offices and competitive compensation.
• Learning and development opportunities, generous holiday allowance and four extra days off for professional development.