Make Smart Decisions Faster: Deciding D2D Resource Allocation via Stackelberg Game Guided Multi-Agent Deep Reinforcement Learning
- 1 June 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Mobile Computing
- Vol. 21 (12) , 4426-4438
Abstract
Device-to-Device (D2D) communication enabling direct data transmission between two mobile users has emerged as a vital component for 5G cellular networks to improve spectrum utilization and enhance system capacity. A critical issue for realizing these benefits in D2D-enabled networks is to properly allocate radio resources while coordinating the co-channel interference in a time-varying communication environment. In this paper, we propose a Stackelberg game (SG) guided multi-agent deep reinforcement learning (MADRL) approach, which allows D2D users to make smart power control and channel allocation decisions in a distributed manner. In particular, we define a crucial Stackelberg Q-value (ST-Q) to guide the learning direction, which can be calculated based on the equilibrium achieved in the Stackelberg game. With the guidance of the Stackelberg equilibrium, our approach converges faster with fewer iterations than the general MADRL method and thereby exhibits better performance in handling the network dynamics. After the initial training, each agent can infer timely D2D resource allocation strategies with distributed execution. Extensive simulations are conducted to validate the efficacy of our proposed scheme in developing timely resource allocation strategies. The results also show that our method outperforms the general MADRL based approach in terms of the average utility, channel capacity, and training time.Keywords
Funding Information
- National Science Foundation of Sri Lanka (CNS-1801925)
- National Science Foundation of Sri Lanka (CNS-2029569)
- JSPS KAKENHI (JP19H02142)
- National Science Foundation of Sri Lanka (EARS-1839818)
- National Science Foundation of Sri Lanka (CNS-1717454)
- National Science Foundation of Sri Lanka (CNS-1731424)
- National Science Foundation of Sri Lanka (CNS-1702850)
- National Science Foundation of Sri Lanka (CCF-1908308)
This publication has 38 references indexed in Scilit:
- User Scheduling and Resource Allocation in HetNets With Hybrid Energy Supply: An Actor-Critic Reinforcement Learning ApproachIEEE Transactions on Wireless Communications, 2017
- A multi-agent reinforcement learning algorithm based on Stackelberg game2017
- Mobile Data Transactions in Device-to-Device Communication Networks: Pricing and AuctionIEEE Wireless Communications Letters, 2016
- Human-level control through deep reinforcement learningNature, 2015
- Joint Mode Selection and Resource Allocation for Device-to-Device CommunicationsIEEE Transactions on Communications, 2014
- Channel Estimation for OFDMIEEE Communications Surveys & Tutorials, 2014
- An overview of 3GPP device-to-device proximity servicesIEEE Communications Magazine, 2014
- Joint routing and link scheduling for cognitive radio networks under uncertain spectrum supply2011
- An overview of limited feedback in wireless communication systemsIEEE Journal on Selected Areas in Communications, 2008
- A Unified Analysis of Value-Function-Based Reinforcement-Learning AlgorithmsNeural Computation, 1999