Model-Based Off-Policy Deep Reinforcement Learning With Model-Embedding
- 12 March 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Emerging Topics in Computational Intelligence
- Vol. 8 (4) , 2974-2986
Abstract
Model-based reinforcement learning (MBRL) has shown its advantages in sample efficiency over model-free reinforcement learning (MFRL) by leveraging control-based domain knowledge. Despite the impressive results it achieves, MBRL is still outperformed by MFRL due to the lack of unlimited interactions with the environment. While imaginary data can be generated by imagining the trajectories of future states, a trade-off between the usage of data generation and the influence of model bias remains to be resolved. In this paper, we propose a simple and elegant off-policy model-based deep reinforcement learning algorithm with a model embedded in the framework of probabilistic reinforcement learning, called MEMB. To balance the sample-efficiency and model bias, we exploit both real and imaginary data in training. In particular, we embed the model in the policy update and learn value functions from the real data set. We also provide a theoretical analysis of MEMB with the Lipschitz continuity assumption on the model and policy, proving the reliability of the short-term imaginary rollout. Finally, we evaluate MEMB on several benchmarks and demonstrate that our algorithm can achieve state-of-the-art performance.Keywords
Funding Information
- National Natural Science Foundation of China (62102241)
- Natural Science Foundation of Shanghai Municipality (23ZR1425400)
This publication has 18 references indexed in Scilit:
- Effective ensemble learning approach for large-scale medical data analyticsInternational Journal of System Assurance Engineering and Management, 2022
- Off-policy Q-learning: Solving Nash equilibrium of multi-player games with network-induced delay and unmeasured stateAutomatica, 2022
- Model-based Constrained Reinforcement Learning using Generalized Control Barrier Function2021
- The Wisdom of the Crowd: Reliable Deep Reinforcement Learning Through Ensembles of Q-FunctionsIEEE Transactions on Neural Networks and Learning Systems, 2021
- An Off-Policy Trust Region Policy Optimization Method With Monotonic Improvement Guarantee for Deep Reinforcement LearningIEEE Transactions on Neural Networks and Learning Systems, 2021
- Robot-Assisted Training in Laparoscopy Using Deep Reinforcement LearningIEEE Robotics and Automation Letters, 2019
- Applications of Deep Learning and Reinforcement Learning to Biological DataIEEE Transactions on Neural Networks and Learning Systems, 2018
- Deep Direct Reinforcement Learning for Financial Signal Representation and TradingIEEE Transactions on Neural Networks and Learning Systems, 2016
- Model-Based Reinforcement Learning for Infinite-Horizon Approximate Optimal TrackingIEEE Transactions on Neural Networks and Learning Systems, 2016
- Human-level control through deep reinforcement learningNature, 2015