Exploring the Ethical Challenges of Conversational AI in Mental Health Care: Scoping Review
Top Cited Papers
Open Access
- 21 February 2025
- journal article
- review article
- Published by JMIR Publications Inc. in JMIR Mental Health
- Vol. 12 (1) , e60432
Abstract
Background: Conversational artificial intelligence (CAI) is emerging as a promising digital technology for mental health care. CAI apps, such as psychotherapeutic chatbots, are available in app stores, but their use raises ethical concerns. Objective: We aimed to provide a comprehensive overview of ethical considerations surrounding CAI as a therapist for individuals with mental health issues. Methods: We conducted a systematic search across PubMed, Embase, APA PsycINFO, Web of Science, Scopus, the Philosopher’s Index, and ACM Digital Library databases. Our search comprised 3 elements: embodied artificial intelligence, ethics, and mental health. We defined CAI as a conversational agent that interacts with a person and uses artificial intelligence to formulate output. We included articles discussing the ethical challenges of CAI functioning in the role of a therapist for individuals with mental health issues. We added additional articles through snowball searching. We included articles in English or Dutch. All types of articles were considered except abstracts of symposia. Screening for eligibility was done by 2 independent researchers (MRM and TS or AvB). An initial charting form was created based on the expected considerations and revised and complemented during the charting process. The ethical challenges were divided into themes. When a concern occurred in more than 2 articles, we identified it as a distinct theme. Results: We included 101 articles, of which 95% (n=96) were published in 2018 or later. Most were reviews (n=22, 21.8%) followed by commentaries (n=17, 16.8%). The following 10 themes were distinguished: (1) safety and harm (discussed in 52/101, 51.5% of articles); the most common topics within this theme were suicidality and crisis management, harmful or wrong suggestions, and the risk of dependency on CAI; (2) explicability, transparency, and trust (n=26, 25.7%), including topics such as the effects of “black box” algorithms on trust; (3) responsibility and accountability (n=31, 30.7%); (4) empathy and humanness (n=29, 28.7%); (5) justice (n=41, 40.6%), including themes such as health inequalities due to differences in digital literacy; (6) anthropomorphization and deception (n=24, 23.8%); (7) autonomy (n=12, 11.9%); (8) effectiveness (n=38, 37.6%); (9) privacy and confidentiality (n=62, 61.4%); and (10) concerns for health care workers’ jobs (n=16, 15.8%). Other themes were discussed in 9.9% (n=10) of the identified articles. Conclusions: Our scoping review has comprehensively covered ethical aspects of CAI in mental health care. While certain themes remain underexplored and stakeholders’ perspectives are insufficiently represented, this study highlights critical areas for further research. These include evaluating the risks and benefits of CAI in comparison to human therapists, determining its appropriate roles in therapeutic contexts and its impact on care access, and addressing accountability. Addressing these gaps can inform normative analysis and guide the development of ethical guidelines for responsible CAI use in mental health care.Keywords
All Related Versions
This publication has 107 references indexed in Scilit:
- AI as a Mental Health Therapist for AdolescentsJAMA Pediatrics, 2023
- The oracle of Delphi 2.0: considering artificial intelligence as a challenging tool for the treatment of eating disordersEating and Weight Disorders - Studies on Anorexia, Bulimia and Obesity, 2023
- Emotional Robotics: Curse or Blessing in Psychiatric Care?2022
- Psychiatrists’ Views on Robot-Assisted Diagnostics of Peripartum Depression2021
- The Implications of Embodied Artificial Intelligence in Mental Healthcare for Digital Wellbeing2020
- Is Big Data the New Stethoscope? Perils of Digital Phenotyping to Address Mental IllnessPhilosophy & Technology, 2020
- Digital Interventions for Mental Disorders: Key Features, Efficacy, and Potential for Artificial Intelligence Applications2019
- Talking to Machines About Personal Mental Health ProblemsJAMA, 2017
- Health Care, Capabilities, and AI Assistive TechnologiesEthical Theory and Moral Practice, 2009
- Software agents and robots in mental therapy: psychological and sociological perspectivesAI & SOCIETY, 2008