Constrained Deep Reinforcement Learning for Energy Sustainable Multi-UAV Based Random Access IoT Networks With NOMA
- 25 August 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal on Selected Areas in Communications
- Vol. 39 (4) , 1101-1115
Abstract
In this paper, we apply the Non-Orthogonal Multiple Access (NOMA) technique to improve the massive channel access of a wireless IoT network where solar-powered Unmanned Aerial Vehicles (UAVs) relay data from IoT devices to remote servers. Specifically, IoT devices contend for accessing the shared wireless channel using an adaptive $p$ -persistent slotted Aloha protocol; and the solar-powered UAVs adopt Successive Interference Cancellation (SIC) to decode multiple received data from IoT devices to improve access efficiency. To enable an energy-sustainable capacity-optimal network, we study the joint problem of dynamic multi-UAV altitude control and multi-cell wireless channel access management of IoT devices as a stochastic control problem with multiple energy constraints. We first formulate this problem as a Constrained Markov Decision Process (CMDP), and propose an online model-free Constrained Deep Reinforcement Learning (CDRL) algorithm based on Lagrangian primal-dual policy optimization to solve the CMDP. Extensive simulations demonstrate that our proposed algorithm learns a cooperative policy in which the altitude of UAVs and channel access probability of IoT devices are dynamically controlled to attain the maximal long-term network capacity while ensuring energy sustainability of UAVs, outperforming baseline schemes. The proposed CDRL agent can be trained on a small network, yet the learned policy can efficiently manage networks with a massive number of IoT devices and varying initial states, which can amortize the cost of training the CDRL agent.Keywords
All Related Versions
Funding Information
- NSF (ECCS-1554576, ECCS-1610874, CNS-1816908)
- U.S. Department of Energy, Office of Science (DE-AC02-06CH11357)
This publication has 32 references indexed in Scilit:
- Optimal Path Planning of Solar-Powered UAV Using Gravitational Potential EnergyIEEE Transactions on Aerospace and Electronic Systems, 2017
- Placement Optimization of UAV-Mounted Mobile Base StationsIEEE Communications Letters, 2016
- Efficient Deployment of Multiple Unmanned Aerial Vehicles for Optimal Wireless CoverageIEEE Communications Letters, 2016
- Perpetual flight with a small solar-powered UAV: Flight results, performance analysis and model validation2016
- Solar powered UAV: Design and experiments2015
- An Online Actor–Critic Algorithm with Function Approximation for Constrained Markov Decision ProcessesJournal of Optimization Theory and Applications, 2012
- An actor–critic algorithm with function approximation for discounted cost constrained Markov decision processesSystems & Control Letters, 2010
- A Survey of Motion Planning Algorithms from the Perspective of Autonomous UAV GuidanceJournal of Intelligent & Robotic Systems, 2009
- An actor-critic algorithm for constrained Markov decision processesSystems & Control Letters, 2005
- Least squares quantization in PCMIEEE Transactions on Information Theory, 1982