The AI Interviewer: Mastering Quality, Rigor, and Deployment in Mixed-Methods Evaluation

Artificial intelligence is rapidly entering evaluation practice, yet there is little guidance on how to judge the quality of AI-mediated interviews. This session presents a theory-grounded framework for assessing AI interview performance using eight operational measures derived from evaluation and qualitative research theory. Drawing on an empirical comparison of six LLM-based interviewers across simulated evaluation scenarios, the presentation highlights capability boundaries, methodological risks, and practical safeguards for evaluators considering AI-mediated data collection.

Participants will learn how to design and deploy an AI-powered interview agent for mixed-methods data collection in evaluation. Using insights from successful field pilots in humanitarian and development sector evaluations, this hands-on session will guide participants to jointly develop an interview guide, define the agent’s parameters, and deploy a live AI-facilitated survey. Participants will take the survey themselves and review real-time results, reflecting on methodological implications, ethical considerations, data quality, and practical applications.