AI in Action: Human-in-the-Loop Evaluation for Rigorous Insights

The session explores two practical AI applications in evaluation through a human-in-the-loop lens, embedding AI across the data analytics phase. The first case demonstrates classifying 1,000+ project documents into interventions using a pre-defined taxonomy via a five-stage pipeline: inductive taxonomy development, LLM-assisted multi-level classification, benchmarking against human validation, and iterative QA to ensure transparency, accuracy, and interpretability. A second segment expands this perspective by introducing geospatial AI as a complementary set of applications for generating evaluative evidence from spatial data. It presents three case studies illustrating the use of AI-generated datasets and AI modelling approaches across different evaluative contexts. Across these applications, the session highlights how AI is not only supporting analysis but actively shaping the data itself. This shift raises critical methodological and ethical considerations, including bias in training data, uneven geographic representation, opacity in model pipelines, and challenges to reproducibility. Emphasis is placed on the role of human-in-the-loop approaches to validate AI-generated outputs, ensure contextual relevance, and maintain accountability in evaluation practice.