STDPG: A Spatio-Temporal Deterministic Policy Gradient Agent for Dynamic Routing in SDN
Open Access
- 1 June 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-6
Abstract
Dynamic routing in software-defined networking (SDN) can be viewed as a centralized decision-making problem. Most of the existing deep reinforcement learning (DRL) agents can address it, thanks to the deep neural network (DNN) incorporated. However, fully-connected feed-forward neural network (FFNN) is usually adopted, where spatial correlation and temporal variation of traffic flows are ignored. This drawback usually leads to significantly high computational complexity due to large number of training parameters. To overcome this problem, we propose a novel model-free framework for dynamic routing in SDN, which is referred to as spatio-temporal deterministic policy gradient (STDPG) agent. Both the actor and critic networks are based on identical DNN structure, where a combination of convolutional neural network (CNN) and long short-term memory network (LSTM) with temporal attention mechanism, CNN-LSTM-TAM, is devised. By efficiently exploiting spatial and temporal features, CNN-LSTM-TAM helps the STDPG agent learn better from the experience transitions. Furthermore, we employ the prioritized experience replay (PER) method to accelerate the convergence of model training. The experimental results show that STDPG can automatically adapt for current network environment and achieve robust convergence. Compared with a number state-of the-art DRL agents, STDPG achieves better routing solutions in terms of the average end-to-end delay.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- Blockchain-Based Software-Defined Industrial Internet of Things: A Dueling Deep ${Q}$ -Learning ApproachIEEE Internet of Things Journal, 2018
- A Survey of Machine Learning Techniques Applied to Software Defined Networking (SDN): Research Issues and ChallengesIEEE Communications Surveys & Tutorials, 2018
- Joint Admission Control and Routing Via Approximate Dynamic Programming for Streaming Video Over Software-Defined NetworkingIEEE Transactions on Multimedia, 2016
- Deep learningNature, 2015
- Human-level control through deep reinforcement learningNature, 2015
- The road to SDNACM SIGCOMM Computer Communication Review, 2014
- Learning from Imbalanced DataIEEE Transactions on Knowledge and Data Engineering, 2009
- Simplifying the synthesis of internet traffic matricesACM SIGCOMM Computer Communication Review, 2005
- Violin Plots: A Box Plot-Density Trace SynergismThe American Statistician, 1998
- Long Short-Term MemoryNeural Computation, 1997