From Pilot to Practice: A Framework for Testing AI Tools in Evaluation
Mesa | En línea
-
Organizado por:
Social Impact
- En alianza con: Millennium Challenge Corporation
Sobre el evento
How do you decide when an AI tool is good enough to use in evaluation? Since 2023, Social Impact has run structured pilots comparing AI tools against manual approaches at multiple stages of the evaluation process. Social Impact will share findings from its most recent pilot with the Millennium Challenge Corporation (MCC), a multi-method comparative study of AI-assisted transcription, translation, and qualitative coding. As evaluation commissioner, MCC will discuss how it weighs the benefits and risks of AI use, what it needs before trusting AI-assisted findings, and how its policies are evolving.
Presentador/a
| Nombre | Título | Biografía |
|---|---|---|
| Dr. Kari Nelson | Technical Director, Social Impact, Inc. | Dr. Kari Nelson is a Technical Director at Social Impact with 25+ evaluations across energy, WASH, agriculture, health, and education. A mixed-methods specialist, she currently leads water and power evaluations and pilots AI for qualitative coding, translation, and transcription. |
| Julian Glucroft | Director, Dept. of Policy and Evaluation, Millennium Challenge Corporation | Julian Glucroft is a Director at MCC specializing in evidence-based project design and complex evaluations. With 15+ years of experience in M&E, policy and institutional reform, and the energy sector, he brings clear insights and practical solutions to development challenges. |
Moderador/a
| Nombre | Título | Biografía |
|---|---|---|
| Mateusz Pucilowski | Chief Executive Officer, Social Impact, Inc. | Mateusz Pucilowski is CEO of Social Impact and an evaluation methodologist with 16+ years' experience. He is leading SI's AI integration initiative, developing systems that improve efficiency while maintaining analytical quality, and has overseen multiple pilots of AI-enhanced evaluation processes. |
Resumen
AI pilots demonstrate that AI can improve efficiency and reduce costs in evaluation workflows, but its effectiveness depends on the task. Transcription and translation tools showed strong performance and generated substantial savings, while AI-assisted de-identification and qualitative coding did not yet achieve the level of accuracy and consistency required to replace human researchers. Across all pilots, the findings highlighted the importance of human review, transparent documentation, data privacy safeguards, and rigorous quality assurance. Overall, AI shows considerable promise as a tool to augment evaluation practice, but continued testing and refinement are needed before it can be relied upon for complex analytical tasks.
Follow-up actions include:
-Continue refining the qualitative coding pilot, including testing additional model configurations and analyzing results across more studies. The team emphasized the pilot is still ongoing and findings are preliminary.
-Scale and publish findings from the AI pilots and continue developing guidance on responsible AI use in evaluation.
-Continue expanding use of AI for transcription and translation in widely spoken languages while piloting performance in additional languages.
Audience Suggestions Raised:
-Offer a French-language version of the presentation for francophone colleagues in West Africa (where most pilots occurred).