Managing a “responsibility vacuum” in AI monitoring and governance in healthcare: a qualitative study
Authors:
Top Cited Papers
Open Access
- 29 September 2025
- journal article
- research article
- Published by Springer Nature in BMC Health Services Research
- Vol. 25 (1) , 1-10
Abstract
Background: Despite the increasing implementation of artificial intelligence (AI) and machine learning (ML) technologies in healthcare, their long-term safety, effectiveness, and equity remain compromised by a lack of sustained oversight. This study explores the phenomenon of a “responsibility vacuum” in AI governance, wherein maintenance and monitoring tasks are poorly defined, inconsistently performed, and undervalued across healthcare systems. Methods: We conducted semi-structured interviews with 21 experts involved in AI implementation in healthcare, including clinicians, clinical informaticists, computer scientists, and legal/policy professionals. Participants were recruited through purposive and snowball sampling. Interviews were transcribed, coded, and analyzed using abductive qualitative methods to identify themes related to maintenance practices, institutional incentives, and responsibility attribution. Results: Participants widely recognized that AI models degrade over time due to factors such as data drift, changes in clinical practice, and poor generalizability. However, monitoring practices remain ad hoc and fragmented, with few institutions investing in structured oversight infrastructure. This “responsibility vacuum” is perpetuated by institutional incentives favoring rapid innovation and strategic ignorance of AI failures. Despite these challenges, some participants described grassroots efforts to monitor and maintain AI systems, drawing inspiration from fields such as radiology, laboratory medicine, and transportation safety. Conclusions: Our findings suggest that institutional and cultural forces in healthcare deprioritize the maintenance of AI tools, creating a governance gap that may lead to patient harm and inequitable outcomes. Addressing this responsibility vacuum will require formalized accountability structures, interdisciplinary collaboration, and policy reforms that center long-term safety and equity. Without such changes, AI/ML technologies designed to improve patient health may introduce new forms of harm, ultimately eroding trust in AI and machine learning for healthcare.Keywords
This publication has 22 references indexed in Scilit:
- Consideration and Disclosure of Group Risks in Genomics and Other Data-Centric Research: Does the Common Rule Need Revision?The American Journal of Bioethics, 2023
- From Medicine to Health: The Proliferation and Diversification of Cultural AuthorityJournal of Health and Social Behavior, 2021
- The Clinician and Dataset Shift in Artificial IntelligenceNew England Journal of Medicine, 2021
- Geographic Distribution of US Cohorts Used to Train Deep Learning AlgorithmsJAMA, 2020
- Dissecting racial bias in an algorithm used to manage the health of populationsScience, 2019
- Potential Biases in Machine Learning Algorithms Using Electronic Health Record DataJAMA Internal Medicine, 2018
- After Innovation, Turn to MaintenanceTechnology and Culture, 2018
- The logic of strategic ignoranceThe British Journal of Sociology, 2012
- Regulatory objectivity and the generation and management of evidence in medicineSocial Science & Medicine, 2006
- An Agenda for STSSocial Studies of Science, 1999