Coding Evaluation Policies at Scale: A Human-AI Calibration Approach to Content Analysis
Seminario web | En línea
Sobre el evento
This session presents a case study of human-AI calibration for large-scale qualitative coding of evaluation policies across 20 countries spanning Africa, Asia-Pacific, the Americas, and Europe. Presenters will share how iterative collaboration between evaluators and a large language model helped scale policy analysis while keeping human judgment central, and facilitated the development of a consolidated taxonomy for evaluation policy. They will also discuss what responsible AI looks like in practice, specifically where it adds value and where it falls short.
Presentador/a
| Nombre | Título | Biografía |
|---|---|---|
| Alana Kinarsky, PhD | Research Analyst | Alana Kinarsky is a Research Analyst at the UCLA School of Education and the owner of A.R.K. Consulting, an independent evaluation consultancy. Her research focuses on evaluation policy, evaluation marketplace, and the intersection of AI and evaluation methodology. |
| Élyse McCall-Thomas | PhD Candidate | Elyse McCall-Thomas is a PhD candidate at the University of Ottawa. Her research examines evaluation theory, policy and practice, with a focus on how formal policy frameworks shape organizational and institutional approaches to learning and accountability. |
| Christina A. Christie, PhD | Wasserman Dean and Professor of Education | Christina A. Christie is the Wasserman Dean of the School of Education and Information Studies at UCLA and a Professor in the Division of Social Research Methodology. Her work focuses on advancing the theories and methods used to facilitate social betterment through evaluation. |
| Leslie Fierro, PhD | Sydney Duder Professor in Program Evaluation | Leslie Fierro is the Sydney Duder Professor in Program Evaluation at the Max Bell School of Public Policy at McGill University. Her work focuses on strengthening organizational and systems-level capacity for commissioning, planning, implementing, and using evaluation. She is a current AEA Board Member-at-Large. |
Resumen
They key objectives of our presentation were to share our process of working with AI to develop an integrated taxonomy for evaluation policy and how we mitigated challenges by adjusting our approach to leverage the strengths of AI. We also wanted to launch our PolicyCoder site and learn from participants what they may see as potential use cases. What we learned is that there could be some very useful applications for this site including supporting organizations that may not have the resources to develop an evaluation policy from scratch. They could use our tool to help them ascertain what elements of existing evaluation policies (particularly as they relate to their country and sector) may be useful.
At the end of our presentation we noted that participants could sign up for early access to our PolicyCoder site and will follow-up with those individuals. We will also follow-up with a few key individuals on their gLOCAL presentations to see if we can learn more about their work and if there are opportunities for sharing and collaboration.