Adaptive and Robust Routing With Lyapunov-Based Deep RL in MEC Networks Enabled by Blockchains
- 29 October 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 8 (4) , 2208-2225
Abstract
The most recent development of the Internet of Things brings massive timely sensitive and bursty data flows. Also, joint optimization on storage, computation, and communication is in need for multiaccess edge computing frameworks. The adaptive network control has been explored using deep reinforcement learning (RL), but it is not sufficient for bursty network traffic flows, especially when the network traffic pattern may change over time. We formulate the routing control in an environment with time-variant link delays as a Lyapunov optimization problem. We identify that there is a tradeoff between optimization performance and modeling accuracy when the propagation delays are included. We propose a novel deep RL (DRL)-based adaptive network routing method to tackle the issues mentioned above. A Lyapunov optimization technique is used to reduce the upper bound of the Lyapunov drift, improving queuing stability in networked systems. By modeling the network traffic pattern using the Markovian arrival process, we show that network routing problems can be modeled as Markov decision processes and value-iteration-based RL methods can be used to solve them. We design a blockchain-based protocol using proof of elapsed time consensus mechanism to ensure a trustworthy network statistics information exchange for the routing framework. Experiment results show that the proposed method can learn a routing policy and adapt to the changing environment. The proposed method outperforms the baseline backpressure method in multiple settings and converges faster than existing methods. Moreover, the DRL module can effectively learn a better estimation of the long-term Lyapunov drift and penalty functions, providing superior results in terms of the backlog size, end-to-end latency, age of information, and throughput. Furthermore, the blockchain-based network statistics exchange can provide the routing framework against malicious nodes. In addition, the proposed model performs well under various topologies, and thus can be used in general cases.Keywords
Funding Information
- National Key Research and Development Program of China (2020YFB1807803, 2020YFB1807800)
- National Natural Science Foundation of China (62071067, 62001054, 61771068)
- Natural Science Foundation of Beijing Municipality (4182041)
- Ministry of Education and China Mobile Joint Fund (MCM20180101)
- Ph.D. Student Short-Term Exchange Program of Beijing University of Posts and Telecommunications
- NSF (EARS-1839818, CNS1717454, CNS-1731424, CNS-1702850)
This publication has 64 references indexed in Scilit:
- A survey of point-based POMDP solversAutonomous Agents and Multi-Agent Systems, 2012
- Stability and Probability 1 Convergence for Queueing Networks via Lyapunov OptimizationJournal of Applied Mathematics, 2012
- Stochastic Network Optimization with Application to Communication and Queueing SystemsSynthesis Lectures on Communication Networks, 2010
- Optimal backpressure routing for wireless networks with multi-receiver diversityAd Hoc Networks, 2009
- A stability analysis with time-delay of primal–dual power control in optical linksAutomatica, 2009
- Modeling and Forecasting Electricity Prices with Input/Output Hidden Markov ModelsIEEE Transactions on Power Systems, 2005
- Adaptive Control Algorithms for Decentralized Optimal Traffic Engineering in the InternetIEEE/ACM Transactions on Networking, 2004
- Modeling IP traffic using the batch Markovian arrival processPerformance Evaluation, 2003
- Power allocation and routing in multibeam satellites with time-varying channelsIEEE/ACM Transactions on Networking, 2003
- Control System Analysis and Design Via the “Second Method” of Lyapunov: I—Continuous-Time SystemsJournal of Basic Engineering, 1960