Multi-Agent Reinforcement Learning With Measured Difference Reward for Multi-Association in Ultra-Dense mmWave Network
Open Access
- 10 November 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 10, 118747-118758
Abstract
Millimeter Wave (mmWave) communication technology is anticipated to play a vital role in meeting the growing demand for the scarce bandwidth in wireless communications. However, mmWave networks are highly susceptible to blockage. Thus, some mitigation techniques, such as multi-connectivity, need to be considered. Densely deploying mmWave base stations (mBSs) to form an ultra-dense network (UDN) also helps. With a mix of different technologies, optimally allocating resources becomes challenging. In this paper, we study mmWave user multi-association in a two-tier heterogeneous ultra-dense network (HetUDN) with a relatively large number of user equipments (UEs). We propose a framework of multi-agent reinforcement learning (MARL) to tackle the complicated optimization problem, leveraging its adaptivity to the communication environment. The proposed scheme considers mmWave beam-division based multi-connectivity and takes advantage of a macro base station (MBS) for indirect cooperation among agents (UEs). In particular, we borrow a credit-assignment technique called difference reward (DR) to deal with a relatively large MARL system with a large action space, which, to the best of our knowledge, is the first time to apply MARL with DR in user association. Furthermore, the proposed schemes are scalable mainly due to fixed observation dimensions and individual actions taken by UEs independently, ensuring that the operation is independent of the numbers of mBSs and UEs. Numerical results suggest that the two MARL schemes with measured DR could achieve a good balance between energy efficiency and QoS outage, and the one using extended DR (EDR) offers additional performance improvement.Keywords
Funding Information
- National Science Foundation (2135275)
This publication has 41 references indexed in Scilit:
- Millimeter-Wave Channel Measurements and Analysis for Statistical Spatial Channel Model in In-Building and Urban Environments at 28 GHzIEEE Transactions on Wireless Communications, 2017
- User Association for Load Balancing in Vehicular Networks: An Online Reinforcement Learning ApproachIEEE Transactions on Intelligent Transportation Systems, 2017
- Performance Analysis of Multiple Association in Ultra-Dense NetworksIEEE Transactions on Communications, 2017
- Modeling and Analyzing Millimeter Wave Cellular SystemsIEEE Transactions on Communications, 2016
- Ultra-Dense Networks: A SurveyIEEE Communications Surveys & Tutorials, 2016
- User-centric ultra-dense networks for 5G: challenges, methodologies, and directionsIEEE Wireless Communications, 2016
- Multiple association in ultra-dense networks2016
- Beam Division Multiple Access Transmission for Massive MIMO CommunicationsIEEE Transactions on Communications, 2015
- A new fairness index for radio resource allocation in wireless networks2005
- Optimal Payoff Functions for Members of Collectives2002