Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications
Open Access
- 4 August 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 4673-4684
Abstract
There is increasing interest in data-driven approaches for recommending optimal treatment strategies in many chronic disease management and critical care applications. Reinforcement learning methods are well-suited to this sequential decision-making problem, but must be trained and evaluated exclusively on retrospective medical record datasets as direct online exploration is unsafe and infeasible. Despite this requirement, the vast majority of treatment optimization studies use off-policy RL methods (e.g., Double Deep Q Networks (DDQN) or its variants) that are known to perform poorly in purely offline settings. Recent advances in offline RL, such as Conservative Q-Learning (CQL), offer a suitable alternative. But there remain challenges in adapting these approaches to real-world applications where suboptimal examples dominate the retrospective dataset and strict safety constraints need to be satisfied. In this work, we introduce a practical and theoretically grounded transition sampling approach to address action imbalance during offline RL training. We perform extensive experiments on two real-world tasks for diabetes and sepsis treatment optimization to compare performance of the proposed approach against prominent off-policy and offline RL baselines (DDQN and CQL). Across a range of principled and clinically relevant metrics, we show that our proposed approach enables substantial improvements in expected health outcomes and in consistency with relevant practice and safety guidelines.Keywords
Funding Information
- A*STAR, Singapore (Grant No. H19/01/a0/023 ? Diabetes Clinic of the Future)
This publication has 14 references indexed in Scilit:
- Transatlantic transferability of a new reinforcement learning model for optimizing haemodynamic treatment for critically ill patients with sepsisArtificial Intelligence in Medicine, 2021
- Basal Glucose Control in Type 1 Diabetes Using Deep Reinforcement Learning: An In Silico ValidationIEEE Journal of Biomedical and Health Informatics, 2020
- A Deep Deterministic Policy Gradient Approach to Medication Dosing and Surveillance in the ICU2018
- Mastering the game of Go without human knowledgeNature, 2017
- Deep reinforcement learning for automated radiation adaptation in lung cancerMedical Physics, 2017
- Mixed Methods SamplingJournal of Mixed Methods Research, 2007
- A large-scale evaluation of features for automatic detection of oil spills in ERS SAR images2005
- Clinical Inertia in Response to Inadequate Glycemic ControlDiabetes Care, 2005
- Weighted Average Importance Sampling and Defensive Mixture DistributionsTechnometrics, 1995
- VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITYMonthly Weather Review, 1950