A Single-Group, Longitudinal Effectiveness Study of an Artificial Intelligence (AI)-Supported Tool for Teaching Spanish-speaking Mental Health Providers the Skill of Reflective Listening (Preprint)
BACKGROUND Mental health services in Latin America face shortages of trained professionals, and access to expertise in evidence-based communication skills remains severely limited. In particular, motivational interviewing (MI) "reflections" are a foundational communication skill for empathic listening, but opportunities for individualized practice with feedback are limited. Digital resources supported by artificial intelligence (AI) may expand opportunities for skill development. OBJECTIVE To evaluate an AI-supported online training tool designed to improve the quality of MI reflections among psychologists, psychology students, and community-based providers in Honduras, and to characterize participants’ experiences with the training. METHODS We conducted a longitudinal single-group study among 56 Spanish-speaking adults in Honduras, including 22 psychologists, 11 psychology students, and 23 community leaders working in mental health service roles. Participants completed four rounds of online exercises in which they generated reflective responses to 10 client statements and received AI-generated scores ranging from 0 to 1, along with examples of high-quality responses. The scoring model used the Prompt-Aware Margin Ranking (PAIR) framework, a RoBERTa-based cross-encoder trained using ordinal categories identifying "skilled" responses, "acceptable" responses, and "MI-inconsistent" responses. Participants also received a live booster session with a psychologist between the second and third rounds. Changes in reflection scores were examined longitudinally, and differences in performance among the three participant groups were assessed. Semi-structured interviews were conducted with 12 purposively selected participants to identify perceived benefits, limitations, and recommendations for improving the intervention. RESULTS All participants completed all four rounds, contributing 4480 reflection responses. Mean scores increased across rounds (P<.001). Improvements in scores were observed among psychologists (P=.009), psychology students (P<.001), and community leaders (P<.001). The proportion of responses classified as MI-skilled increased from 26.3% in the first round to 48.2% in the fourth (P<.001), while MI-inconsistent responses decreased from 42.4% to 22.7% (P<.001). Improvements were greatest among community leaders, whose skilled reflections increased from 5.4% to 37.7%. The live booster was associated with significant score improvements over-and-above the linear trend (P<.001), with notable benefit among community leaders (P=.002). Interview participants generally described the tool as engaging, useful, and helpful for translating knowledge about reflective listening into practice. Participants valued both the automated feedback and live booster session, while recommending greater cultural and contextual adaptation of scenarios to Honduran experiences. CONCLUSIONS An AI-supported training platform combined with brief human reinforcement was associated with substantial short-term improvements in the quality of MI reflections among psychologists, psychology students, and community-based providers in Honduras. The largest gains occurred among participants without formal training in clinical psychology, suggesting that the tool could improve the quality of interactions in environments relying on paraprofessionals. Future controlled studies should determine whether improvements are sustained and translate into changes in communication with clients and subsequent clinical outcomes. CLINICALTRIAL Not applicable