On the Challenges of Using Reinforcement Learning in Precision Drug Dosing: Delay and Prolongedness of Action Effects
Open Access
- 26 June 2023
- journal article
- research article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 37 (12) , 14102-14109
Abstract
Drug dosing is an important application of AI, which can be formulated as a Reinforcement Learning (RL) problem. In this paper, we identify two major challenges of using RL for drug dosing: delayed and prolonged effects of administering medications, which break the Markov assumption of the RL framework. We focus on prolongedness and define PAE-POMDP (Prolonged Action Effect-Partially Observable Markov Decision Process), a subclass of POMDPs in which the Markov assumption does not hold specifically due to prolonged effects of actions. Motivated by the pharmacology literature, we propose a simple and effective approach to converting drug dosing PAE-POMDPs into MDPs, enabling the use of the existing RL algorithms to solve such problems. We validate the proposed approach on a toy task, and a challenging glucose control task, for which we devise a clinically-inspired reward function. Our results demonstrate that: (1) the proposed method to restore the Markov assumption leads to significant improvements over a vanilla baseline; (2) the approach is competitive with recurrent policies which may inherently capture the prolonged affect of actions; (3) it is remarkably more time and memory efficient than the recurrent baseline and hence more suitable for real-time dosing control systems; and (4) it exhibits favourable qualitative behavior in our policy analysis.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Optimizing warfarin dosing using deep reinforcement learningJournal of Biomedical Informatics, 2022
- Reinforcement learning application in diabetes blood glucose control: A systematic reviewArtificial Intelligence in Medicine, 2020
- Model‐Informed Artificial Intelligence: Reinforcement Learning for Precision DosingClinical Pharmacology & Therapeutics, 2020
- Deep Reinforcement Learning for Optimal Critical Care Pain Management with Morphine using Dueling Double-Deep Q Networks2019
- Pharmacodynamic principles and the time course of delayed and cumulative drug effectsTranslational and Clinical Pharmacology, 2018
- Mechanisms of Drug Toxicity and Relevance to Pharmaceutical DevelopmentDrug Metabolism and Pharmacokinetics, 2011
- In Silico Preclinical Trials: A Proof of Concept in Closed-Loop Control of Type 1 DiabetesJournal of Diabetes Science and Technology, 2009
- Exponential Growth and Decay2007
- Drug Concentration, Binding, and Effect In VivoPharmaceutical Research, 1984