Deep Reinforcement Learning for Dynamic Spectrum Access: Convergence Analysis and System Design
- 5 August 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Wireless Communications
- Vol. 23 (12) , 18888-18902
Abstract
In dynamic spectrum access (DSA) networks, secondary users (SUs) need to opportunistically access primary users’ (PUs) radio spectrum without causing significant interference. Since the SU-PU interaction is limited, deep reinforcement learning has been introduced to help SUs conduct spectrum access. Specifically, deep recurrent Q network (DRQN) has been utilized in DSA networks for SUs to aggregate information from recent experiences to make spectrum access decisions. DRQN is notorious for its sample efficiency since it needs a rather large number of training samples to tune its parameters which is a computationally demanding task. Deep echo state network (DEQN) has been introduced for DSA networks to address the sample efficiency issue of DRQN. In this work, we compare the convergence of DRQN and DEQN by comparing the upper bounds we obtain on their covering number, a notion of richness. Furthermore, we introduce a method to determine the right hyper-parameters for DEQN, providing system design guidance for DEQN-based DSA networks. Extensive performance evaluation confirms that DEQN-based DSA strategy is the superior choice with regard to computational power while outperforming DRQN-based ones.Keywords
Funding Information
- U.S. National Science Foundation (CNS-1811720, CCF-1937487, ECCS-2128594)
This publication has 25 references indexed in Scilit:
- DDPG-Based Joint Time and Energy Management in Ambient Backscatter-Assisted Hybrid Underlay CRNsIEEE Transactions on Communications, 2022
- Deep Echo State Q-Network (DEQN) and Its Application in Dynamic Spectrum Sharing for 5G and BeyondIEEE Transactions on Neural Networks and Learning Systems, 2020
- Artificial Intelligence-Enabled Cellular Networks: A Critical Path to Beyond-5G and 6GIEEE Wireless Communications, 2020
- Recent advances in physical reservoir computing: A reviewNeural Networks, 2019
- Distributive Dynamic Spectrum Access Through Deep Reinforcement Learning: A Reservoir Computing-Based ApproachIEEE Internet of Things Journal, 2018
- A Deep Hierarchical Reinforcement Learning Algorithm in Partially Observable Markov Decision ProcessesIEEE Access, 2018
- Design and Implementation of VoIP Based Two-Tier Cognitive Radio Network for Improved Spectrum UtilizationIEEE Systems Journal, 2015
- Lessons Learned from an Extensive Spectrum Occupancy Measurement Campaign and a Stochastic Duty Cycle ModelMobile Networks and Applications, 2009
- On myopic sensing for multi-channel opportunistic access: structure, optimality, and performanceIEEE Transactions on Wireless Communications, 2008
- Random waypoint mobility model in cellular networksWireless Networks, 2006