A chatbot for the soul: mental health care, privacy, and intimacy in AI-based conversational agents
Open Access
- 29 October 2025
- journal article
- research article
- Published by Springer Nature in Communication and Change
- Vol. 1 (1) , 1-23
Abstract
Artificial intelligence-based conversational agents—chatbots—are increasingly integrated into telehealth platforms, employee wellness programs, and mobile applications to address structural gaps in mental health care. While these chatbots promise accessibility, they are often deployed without sufficient impact assessment or even basic user testing. This paper presents a case study using community red-teaming exercises to evaluate a chatbot designed for wellness and spirituality. Unlike traditional red-teaming, which is conducted by engineers to assess vulnerabilities, community red-teaming treats impacted users as experts, uncovering concerns related to privacy, ethics, and functionality. Our fieldwork, conducted with undergraduate beta testers ( n = 28), revealed that participants were often more comfortable sharing private information with the chatbot than with a stranger. Prior experience with commercial AI systems, such as ChatGPT, contributed to this ease. However, participants also raised concerns about misinformation, inadequate guardrails for sensitive topics, data security, and dependency. Despite these risks, users remained open to the chatbot’s potential as a spiritual wellness guide. We further examine how algorithmic impact assessments (AIAs) both capture and overlook key aspects of the spiritual chatbot user experience. These chatbots offer hyper-personalized, AI-mediated divination and wellness interactions, blurring the boundaries between astrology, mental health support, and spiritual guidance. The chameleon-like nature of these technologies challenges conventional assessment frameworks, necessitating a nuanced approach that considers specific use cases, potential user bases, and indirect impacts on broader communities. We argue that AIAs for care and wellness chatbots must account for these complexities, ensuring ethical deployment and mitigating harm.Keywords
Funding Information
- Mozilla Foundation
- Open Society Foundations
This publication has 54 references indexed in Scilit:
- Human-AI interaction research agenda: A user-centered perspectiveData and Information Management, 2024
- A sociotechnical system perspective on AIMinds and Machines, 2024
- The therapeutic effectiveness of artificial intelligence-based chatbots in alleviation of depressive and anxiety symptoms in short-course treatments: A systematic review and meta-analysisJournal of Affective Disorders, 2024
- AI Chatbots, Health Privacy, and Challenges to HIPAA ComplianceJAMA, 2023
- Use of Psychics for Stress and Emotional Problems: A Descriptive Survey Comparison with Conventional Providers and Informal HelpersAdministration and Policy in Mental Health and Mental Health Services Research, 2021
- My Chatbot Companion - a Study of Human-Chatbot RelationshipsInternational Journal of Human-Computer Studies, 2021
- On the importance of ethnographic methods in AI researchNature Machine Intelligence, 2021
- Chatbots: History, technology, and applicationsMachine Learning with Applications, 2020
- “Blessed by the algorithm”: Theistic conceptions of artificial intelligence in online discourseAI & SOCIETY, 2020
- The computational therapeutic: exploring Weizenbaum’s ELIZA as a history of the presentAI & SOCIETY, 2018