Hierarchical Reinforcement Learning for Relay Selection and Power Optimization in Two-Hop Cooperative Relay Network
- 13 October 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Communications
- Vol. 70 (1) , 171-184
Abstract
In this paper, we study the outage probability minimizing problem in a two-hop cooperative relay network. To reduce outage probability, existing studies propose many schemes for relay selection and power allocation, which are usually based on the assumption of exact channel state information (CSI). However, it is difficult to obtain perfect instantaneous CSI in practical situations where channel states change rapidly, and thus traditional methods would not perform well. Considering these factors, we turn to the emerging reinforcement learning (RL) methods for solutions. RL methods do not need any prior knowledge of CSI, but use neural network for approximation and decision after interacting with communication environment. Nevertheless, conventional RL methods, including most deep reinforcement learning (DRL) methods, cannot perform well when the search space is too large. In addition, non-stationarity is a common problem when using hierarchical reinforcement learning (HRL), which is caused by the changing behavior in different hierarchies. Therefore, we first propose a DRL framework with an outage-based reward function, which is then used as a baseline. Then, we further design an HRL framework and training algorithm. By decomposing relay selection and power allocation into two hierarchical optimization objectives, and combining on- policy and off-policy methods in the HRL framework, our method successfully address the sparse reward and non-stationary problem. Simulation results reveal that compared with traditional DRL method, the proposed HRL training algorithm can converge faster and reduce the outage probability by 8% in two-hop relay network with the same outage threshold.Keywords
All Related Versions
Funding Information
- National Science Foundation of China (42171404, 61771345)
- Science and Technology Commission of Shanghai Municipality (19511102002)
- Shanghai Science and Technology Innovation Action Plan Project (21220713100)
This publication has 35 references indexed in Scilit:
- A Cooperative Multihop Transmission Scheme for Two-Way Amplify-and-Forward Relay NetworksIEEE Transactions on Vehicular Technology, 2017
- Relay aided smart meter to smart meter communication in a microgrid2016
- Reliability Assessment of Protective Relays in Harmonic-Polluted Power SystemsIEEE Transactions on Power Delivery, 2016
- Power Allocation and Relay Selection for AF Cooperative Relay Systems With Imperfect Channel EstimationIEEE Transactions on Vehicular Technology, 2015
- Human-level control through deep reinforcement learningNature, 2015
- Energy-Efficient Power Control for Multiple-Relay Cooperative Networks Using $Q$ -LearningIEEE Transactions on Wireless Communications, 2014
- Statistical channel knowledge-based optimum power allocation for relaying protocols in the high SNR regimeIEEE Journal on Selected Areas in Communications, 2007
- A simple Cooperative diversity method based on network path selectionIEEE Journal on Selected Areas in Communications, 2006
- Symbol error probabilities for general Cooperative linksIEEE Transactions on Wireless Communications, 2005
- Multihop Diversity in Wireless Relaying ChannelsIEEE Transactions on Communications, 2004