Constrained Deep Reinforcement Learning for Energy Sustainable Multi-UAV based Random Access IoT Networks with NOMA
Preprint
Abstract
In this paper, we apply the Non-Orthogonal Multiple Access (NOMA) technique to improve the massive channel access of a wireless IoT network where solar-powered Unmanned Aerial Vehicles (UAVs) relay data from IoT devices to remote servers. Specifically, IoT devices contend for accessing the shared wireless channel using an adaptive $p$-persistent slotted Aloha protocol; and the solar-powered UAVs adopt Successive Interference Cancellation (SIC) to decode multiple received data from IoT devices to improve access efficiency. To enable an energy-sustainable capacity-optimal network, we study the joint problem of dynamic multi-UAV altitude control and multi-cell wireless channel access management of IoT devices as a stochastic control problem with multiple energy constraints. To learn an optimal control policy, we first formulate this problem as a Constrained Markov Decision Process (CMDP), and propose an online model-free Constrained Deep Reinforcement Learning (CDRL) algorithm based on Lagrangian primal-dual policy optimization to solve the CMDP. Extensive simulations demonstrate that our proposed algorithm learns a cooperative policy among UAVs in which the altitude of UAVs and channel access probability of IoT devices are dynamically and jointly controlled to attain the maximal long-term network capacity while maintaining energy sustainability of UAVs. The proposed algorithm outperforms Deep RL based solutions with reward shaping to account for energy costs, and achieves a temporal average system capacity which is $82.4\%$ higher than that of a feasible DRL based solution, and only $6.47\%$ lower compared to that of the energy-constraint-free system.
Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Deep Reinforcement Learning for Minimizing Age-of-Information in UAV-Assisted Networks2019
- Sustainable Wireless IoT Networks With RF Energy Charging Over Wi-Fi (CoWiFi)IEEE Internet of Things Journal, 2019
- Reinforcement Learning for Decentralized Trajectory Design in Cellular UAV Networks With Sense-and-Send ProtocolIEEE Internet of Things Journal, 2018
- Throughput Maximization in Multi-UAV Enabled Communication Systems With Difference ConsiderationIEEE Access, 2018
- Deployment Algorithms for UAV Airborne Networks Toward On-Demand CoverageIEEE Journal on Selected Areas in Communications, 2018
- LoRa technology MAC layer operations and Research issuesProcedia Computer Science, 2018
- NOMA-Based Random Access With Multichannel ALOHAIEEE Journal on Selected Areas in Communications, 2017
- An Online Actor–Critic Algorithm with Function Approximation for Constrained Markov Decision ProcessesJournal of Optimization Theory and Applications, 2012
- An actor–critic algorithm with function approximation for discounted cost constrained Markov decision processesSystems & Control Letters, 2010
- An actor-critic algorithm for constrained Markov decision processesSystems & Control Letters, 2005