Unlocking LLMs: Addressing Scarce Data and Bias Challenges in Mental Health
Preprint
Abstract
Large language models (LLMs) have shown promising capabilities in healthcare analysis but face several challenges like hallucinations, parroting, and bias manifestation. These challenges are exacerbated in complex, sensitive, and low-resource domains. Therefore, in this work we introduce IC-AnnoMI, an expert-annotated motivational interviewing (MI) dataset built upon AnnoMI by generating in-context conversational dialogues leveraging LLMs, particularly ChatGPT. IC-AnnoMI employs targeted prompts accurately engineered through cues and tailored information, taking into account therapy style (empathy, reflection), contextual relevance, and false semantic change. Subsequently, the dialogues are annotated by experts, strictly adhering to the Motivational Interviewing Skills Code (MISC), focusing on both the psychological and linguistic dimensions of MI dialogues. We comprehensively evaluate the IC-AnnoMI dataset and ChatGPT's emotional reasoning ability and understanding of domain intricacies by modeling novel classification tasks employing several classical machine learning and current state-of-the-art transformer approaches. Finally, we discuss the effects of progressive prompting strategies and the impact of augmented data in mitigating the biases manifested in IC-AnnoM. Our contributions provide the MI community with not only a comprehensive dataset but also valuable insights for using LLMs in empathetic text generation for conversational therapy in supervised settings.Keywords
All Related Versions
This publication has 42 references indexed in Scilit:
- Efficient healthcare with large language models: optimizing clinical workflow and enhancing patient careJournal of the American Medical Informatics Association, 2024
- Using cognitive psychology to understand GPT-like models needs to extend beyond human biasesProceedings of the National Academy of Sciences, 2023
- Probing the psychology of AI modelsProceedings of the National Academy of Sciences, 2023
- Beyond bias and discrimination: redefining the AI ethics principle of fairness in healthcare machine-learning algorithmsAI & SOCIETY, 2022
- The Efficacy of Motivational Interviewing in Adults With Chronic Pain: A Meta-Analysis and Systematic ReviewThe Journal of Pain, 2016
- From counselor skill to decreased marijuana use: Does change talk matter?Journal of Substance Abuse Treatment, 2014
- Motivational interviewing in medical care settings: A systematic review and meta-analysis of randomized controlled trialsPatient Education and Counseling, 2013
- From in-session behaviors to drinking outcomes: A causal chain for motivational interviewing.Journal of Consulting and Clinical Psychology, 2009
- Counselor skill influences outcomes of brief motivational interventionsJournal of Substance Abuse Treatment, 2009
- Review of Motivational Interviewing in promoting health behaviorsClinical Psychology Review, 2009