Accelerating Model-Free Reinforcement Learning With Imperfect Model Knowledge in Dynamic Spectrum Access
- 16 April 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 7 (8) , 7517-7528
Abstract
Current studies that apply reinforcement learning (RL) to dynamic spectrum access (DSA) problems in wireless communications systems are mainly focusing on model-free RL. However, in practice model-free RL requires large number of samples to achieve good performance making it impractical in real time applications such as DSA. Combining model-free and model-based RL can potentially reduce the sample complexity while achieving similar level of performance as model-free RL as long as the learned model is accurate enough. However, in complex environment the learned model is never perfect. In this paper we combine model-free and model-based reinforcement learning, introduce an algorithm that can work with an imperfectly learned model to accelerate the model-free reinforcement learning. Results show our algorithm achieves higher sample efficiency than standard model-free RL algorithm and Dyna algorithm (a standard algorithm that integrating model-based and model-free RL) with much lower computation complexity than the Dyna algorithm. For the extreme case where the learned model is highly inaccurate, the Dyna algorithm performs even worse than the model-free RL algorithm while our algorithm can still outperform the model-free RL algorithm.Keywords
Funding Information
- U.S. National Science Foundation (ECCS-1811497, CCF-1937487)
This publication has 20 references indexed in Scilit:
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- Human-level control through deep reinforcement learningNature, 2015
- Scenarios for 5G mobile and wireless communications: the vision of the METIS projectIEEE Communications Magazine, 2014
- Reinforcement learning in robotics: A surveyThe International Journal of Robotics Research, 2013
- Synthesis and stabilization of complex behaviors through online trajectory optimization2012
- Cooperative Q-learning for multiple secondary users in dynamic spectrum access2011
- A Survey on Policy Search for RoboticsFoundations and Trends in Robotics, 2011
- Decentralized cognitive MAC for opportunistic spectrum access in ad hoc networks: A POMDP frameworkIEEE Journal on Selected Areas in Communications, 2007
- Simple statistical gradient-following algorithms for connectionist reinforcement learningMachine Learning, 1992
- Dyna, an integrated architecture for learning, planning, and reactingACM SIGART Bulletin, 1991