Evaluating AI for Early-Career STEM Support: Trust, Use, and Perceived Outcomes from a Chatbot Intervention

This session presents an evaluation of TrueNorth, an AI-powered chatbot designed to provide early-career guidance to STEM professionals. The evaluation was conducted through an interdisciplinary collaboration with active involvement from program partners throughout the design and implementation process. Using a mixed-methods, pre/post design, the evaluation examines participants’ trust in AI, their likelihood of using AI for work-related tasks, and perceived outcomes related to confidence, agency, and access to support. The session will present key findings alongside qualitative insights into how participants perceive and use AI-generated guidance. Beyond reporting results, this session highlights what it means to evaluate AI-based interventions in practice. It will discuss methodological considerations, including working with evolving systems, measuring trust and perceived usefulness, and navigating the role of evaluators in technology development contexts. The session concludes with implications for evaluation practice, focusing on how AI tools are reshaping expectations around evidence, support, and decision-making.

This panel examines the development of TrueNorth, an AI-powered chatbot designed to support early-career STEM professionals, as a case study in what it means to integrate evaluation into AI system design. Emerging from a collaboration between computer science and evaluation students, TrueNorth was developed as a retrieval-augmented generation (RAG) system grounded in a positive psychology framework. Unlike traditional evaluation models, where evaluators assess completed interventions, this project required evaluators to engage during system development, translating constructs such as trust, support, and professional agency into features that could be operationalized within a chatbot environment. Panelists will reflect on key tensions encountered throughout the process, including: 1) how evaluative concepts are simplified, distorted, or constrained when translated into AI systems 2) the gap between what evaluators aim to measure and what systems are designed to produce 3) the challenge of aligning technical feasibility with evaluative rigor how user needs identified through evaluation shaped (and sometimes conflicted with) system design decisions Drawing on experiences from a hackathon, iterative prototyping, and the development of a supporting white paper, this session moves beyond describing collaboration to critically examine what is gained, and lost, when evaluation becomes embedded in AI development. The discussion situates this work within broader shifts in the social sector, where AI tools are increasingly deployed in contexts requiring judgment, care, and contextual understanding. It argues that evaluators must move upstream in the development process -- not only to assess AI systems, but to shape how they define problems, generate outputs, and influence decision-making.