Does Reinforcement Learning Improve Outcomes for Critically Ill Patients? A Systematic Review and Level-of-Readiness Assessment
Authors:
- 8 November 2023
- journal article
- review article
- Published by Wolters Kluwer Health in Critical Care Medicine
- Vol. 52 (2) , e79-e88
Abstract
OBJECTIVE: Reinforcement learning (RL) is a machine learning technique uniquely effective at sequential decision-making, which makes it potentially relevant to ICU treatment challenges. We set out to systematically review, assess level-of-readiness and meta-analyze the effect of RL on outcomes for critically ill patients. DATA SOURCES: A systematic search was performed in PubMed, Embase.com, Clarivate Analytics/Web of Science Core Collection, Elsevier/SCOPUS and the Institute of Electrical and Electronics Engineers Xplore Digital Library from inception to March 25, 2022, with subsequent citation tracking. DATA EXTRACTION: Journal articles that used an RL technique in an ICU population and reported on patient health-related outcomes were included for full analysis. Conference papers were included for level-of-readiness assessment only. Descriptive statistics, characteristics of the models, outcome compared with clinician’s policy and level-of-readiness were collected. RL-health risk of bias and applicability assessment was performed. DATA SYNTHESIS: A total of 1,033 articles were screened, of which 18 journal articles and 18 conference papers, were included. Thirty of those were prototyping or modeling articles and six were validation articles. All articles reported RL algorithms to outperform clinical decision-making by ICU professionals, but only in retrospective data. The modeling techniques for the state-space, action-space, reward function, RL model training, and evaluation varied widely. The risk of bias was high in all articles, mainly due to the evaluation procedure. CONCLUSION: In this first systematic review on the application of RL in intensive care medicine we found no studies that demonstrated improved patient outcomes from RL-based technologies. All studies reported that RL-agent policies outperformed clinician policies, but such assessments were all based on retrospective off-policy evaluation.Keywords
This publication has 40 references indexed in Scilit:
- An automated approach to identifying search terms for systematic reviews using keyword co‐occurrence networksMethods in Ecology and Evolution, 2019
- Guidelines for reinforcement learning in healthcareNature Medicine, 2019
- PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model StudiesAnnals of Internal Medicine, 2019
- An Introduction to Deep Reinforcement LearningFoundations and Trends in Machine Learning, 2018
- The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive careNature Medicine, 2018
- Rayyan—a web and mobile app for systematic reviewsSystematic Reviews, 2016
- Optimal medication dosing from suboptimal clinical examples: A deep reinforcement learning approach2016
- MIMIC-III, a freely accessible critical care databaseScientific Data, 2016
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- From Data to Optimal Decision Making: A Data-Driven, Probabilistic Machine Learning Approach to Decision Support for Patients With SepsisJMIR Medical Informatics, 2015