Yang Xiao

Routing optimization remains challenging in traffic engineering as increasing traffic in dynamic networks often leads to congestion and resource wastage. To address it, probabilistic routing strategies based on deep reinforcement learning (DRL) with finer-grained routing control and autonomous learning capabilities are considered promising solutions. However, current DRL-based probabilistic routing approaches are limited by the lack of constraints for routing safety and cross-comparisons with other routing strategies. This paper proposes a constrained multi-agent DRL-based probabilistic routing approach, which employs a novel actor-critic architecture called JointGAT based on graph attention networks and is trained using clipped proximal policy optimization. Besides, this paper proposes a novel probabilistic routing protocol for constraints to ensure routing safety. For validation and comparison, an open-source general packet-level network simulation framework called RouterRL is designed and implemented based on the knowledge-defined networking architecture, supporting cross-comparisons of different routing strategies. We compare the performance of the proposed approach with various benchmark approaches using different routing strategies. Extensive simulation results across various topologies and loads demonstrate that the proposed approach significantly outperforms benchmark approaches regarding network quality of service and robustness.

By integrating the software-defined networking (SDN) architecture with the machine learning-based knowledge plane, knowledge-defined networking (KDN) is revolutionizing established traffic engineering (TE) methodologies. This paper investigates the challenging joint routing and caching problem in KDN-based networks, managing multiple traffic flows to improve long-term quality-of-service (QoS) performance. This challenge is formulated as a computationally expensive non-convex mixed-integer non-linear programming (MINLP) problem, which exceeds the capacity of heuristic methods to achieve near-optimal solutions. To address this issue, we present DRL-JRC, an actor-critic deep reinforcement learning (DRL) algorithm for adaptive joint routing and caching in KDN-based networks. DRL-JRC orchestrates the optimization of multiple QoS metrics, including end-to-end delay, packet loss rate, load balancing index, and hop count. During offline training, DRL-JRC employs proximal policy optimization (PPO) to smooth the policy optimization process. In addition, the learned policy can be seamlessly integrated with conventional caching solutions during online execution. Extensive experiments demonstrate the comprehensive superiority of DRL-JRC over baseline methods in various scenarios. Meanwhile, DRL-JRC consistently outperforms the heuristic baseline under partial policy deployment during execution. Compared to the average performance of the baseline methods, DRL-JRC reduces the end-to-end delay by 51.14% and the packet loss rate by 40.78%.

The Third Generation Partnership Project has standardized new-radio vehicle-to-everything (NR-V2X) to facilitate advanced use cases for safety-critical message conveyance. However, there is a paucity of resource allocation research for high-performance NR-V2X communications. In this article, we investigate an adaptive optimization issue of energy efficiency (EE) and Quality of Service (QoS) in NR-V2X networks inspired by the Tchebycheff function. To address this issue, we first formulate the resource allocation task as a time-variant mixed-integer nonlinear programming (MINLP) problem. Then, we propose a fully decentralized multiagent deep reinforcement learning (MADRL)-based algorithm characterized by a multitask-actor shared-critic (MTA-SC) architecture and a localized training and distributed execution (LTDE) framework to promote efficient learning and minimize information exchange. Finally, we implement the proposed algorithm in a three-lane highway NR-V2X scenario. Numerical results demonstrate that the proposed algorithm comprehensively outperforms benchmarks in terms of convergence performance, scalability, and robustness.

Multipath routing remains a challenging issue in traffic engineering (TE) as existing solutions are incapable of handling the evolving network dynamics and stringent quality-of-service (QoS) requirements. To address it, multiagent deep reinforcement learning (MADRL) is a promising technique that provides more elaborate multipath routing strategies. However, prevalent MADRL-based solutions still suffer inapplicability as they fail to ensure both scalability and QoS awareness. In this paper, we leverage the emerging hybrid knowledge-defined networking (KDN) architecture, and propose a collaborative MADRL-based multipath routing algorithm. Two novel mechanisms, i.e., parallel agent replication and periodic policy synchronization, are devised for agent design to ensure the practicality of the proposed method. In addition, an efficient communication mechanism is established to facilitate multiagent collaboration by enabling scalable observation and reward exchange. Featuring a multiagent twin-actor-critic (MA-TAC) learning structure and a proximal policy optimization (PPO) -based training process, the proposed algorithm consists of alternately scheduled execution and training phases for practical deployment. We compare the performance of our proposed method with those of several benchmark methods. Extensive simulation results demonstrate that the proposed method achieves significantly better scalability, QoS awareness, and stability than the benchmark methods under various environment settings.

Mobile edge computing (MEC) is an enabling technology for next-generation network architectures to deliver more diversified communication services and meet more demanding quality-of-service (QoS) requirements. However, owing to the growing scale and heterogeneity of the networks, energy-efficient resource allocation in heterogeneous MEC (Het-MEC) networks faces great challenges. As an emerging research area, multi-agent deep reinforcement learning (MADRL) is expected to realize autonomous resource allocation in Het-MEC networks by learning from trial and error. Nevertheless, existing MADRL-based solutions are usually not applicable to practical scenarios by the limitations of centralized control schemes or massive signaling overhead. To address the issue, we first formulate the energy-efficient resource allocation problem in Het-MEC networks as a time-variant mixed-integer nonlinear programming (MINLP) problem. Then, we propose a fully decentralized collaborative MADRL-based algorithm featuring the multi-actor shared-critic (MASC) architecture and the regional training distributed execution (RTDE) framework, which effectively stabilizes model training and reduces information exchange, respectively. Finally, we conduct extensive simulations to evaluate the performance of the proposed algorithm. Numerical results demonstrate that the proposed algorithm comprehensively outperforms several mainstream baseline methods in terms of convergence performance, scalability, and robustness.

As the rising complexity for network simulation and the development of deep reinforcement learning (DRL), existing network simulation frameworks cannot meet the high demand for DRL-based simulation. To solve this issue, we propose RL4NeT++, a packet-level network simulation framework based on OMNeT++ for DRL-based routing algorithms. To the best of our knowledge, RL4NeT++ is the first network simulation framework that takes asynchronous updating strategy into consideration and applies it for consistence to the training logic of DRL. In addition, RL4NeT++ has fast simulation and training speed, user-friendly interface, diverse network configurations, high modularization, convenient customization, and strong adaptability. We run extensive experiments to demonstrate the application of RL4NeT++ in DRL-based routing algorithm and prove the high simulation speed and strong adaptability of RL4NeT++.

As a promising technology in the 5G era, the artificial intelligence (AI) enabled Internet of controllable things (IoCT) is expected to be an integral part of heterogeneous networks (HetNets) in the future. However, the realization of ultra-reliable low-latency communications (URLLC) in IoCT communications underlaid HetNet has stringent quality of service (QoS) requirements, resulting in unprecedented challenges for existing wireless resource allocation methods. In this paper, we first describe a cellular HetNet model with uplink IoCT communications, then formulate a dynamic mixed-integer nonlinear programming (MINLP) resource allocation problem for maximizing the long-term average energy efficiency under URLLC requirements including reliability, latency, and transmission rate. To solve the problem, we propose a decentralized MADRL-based resource allocation algorithm with a decentralized partially observable Markov decision process (dec-POMDP) and a mixed-centralized-decentralized (MCD) framework to address the partial observability and the scalability issues, respectively. In addition, we design a reward function featuring the objective decomposition, baseline-guided scaling, and QoS violation penalty so that the agents are coordinated. Extensive experiments demonstrate the convergence, scalability, and robustness of the proposed algorithm. Besides, the proposed algorithm substantially outperforms conventional resource allocation methods and different agent communication mechanisms in terms of maximizing energy efficiency.

Cognitive routing is expected to be a key technology to realize intellectualized operation and administration for autonomous networks. Meanwhile, reinforcement learning (RL) excels especially in decision-making tasks by learning from trial and error, which is promising for solving various cognitive routing problems. Nevertheless, the research of RL-based cognitive routing has been severely constrained by a lack of general problem formulation and feasible network simulation frameworks. In this letter, we propose the first general problem formulation of RL-based cognitive routing. In addition, based on OpenAI Gym and ns-3, we implement the first packet-level network simulation framework, the RL4Net, that facilitates the rapid prototyping of RL-based cognitive routing algorithms. Finally, we conduct extensive experiments with the RL4Net to evaluate the effectiveness of RL-based cognitive routing. Numerical results demonstrate that our proposed RL-based cognitive routing effectively outperforms RL-based non-cognitive routing and the OSPF in terms of minimizing average end-to-end network delay.

After decades of unprecedented development, modern networks have evolved far beyond expectations in terms of scale and complexity. In many cases, traditional traffic engineering (TE) approaches fail to address the quality of service (QoS) requirements of modern networks. In recent years, deep reinforcement learning (DRL) has proved to be a feasible and effective solution for autonomously controlling and managing complex systems. Massive growth in the use of DRL applications in various domains is beginning to benefit the communications industry. In this paper, we firstly provide a comprehensive overview of DRL-based TE. Then, we present a detailed literature review on applications of DRL for TE including three fundamental issues: routing optimization, congestion control, and resource management. Finally, we discuss our insights into the challenges and future research perspectives of DRL-based TE.

This letter investigates a deep reinforcement learning (DRL)-based spectrum access scheme for device-to-device (D2D) communication underlay cellular networks. Specifically, cellular users (CUEs) and D2D pairs attempt to access the time slots (TSs) of a shared spectrum, and TSs are dynamically scheduled to CUEs in different frames. Based on the DRL theory, D2D pairs can be seen as a centralized agent which aims to learn an optimal spectrum access strategy to maximize the sum throughput without any prior information. In particular, with different locations of CUEs, the spectrum access manners for D2D communication are changed to ensure the communication quality of CUEs at the cell edge. Then, a double deep Q-network (DDQN) based D2D spectrum access (D ⁴ SA) algorithm is proposed, which makes D2D pairs learn to decide whether to access the spectrum in different TSs. Moreover, to ensure the fairness of resource allocation among D2D pairs, we improve the proposed algorithm and incorporate fairness into the objective function. Simulation results show that our proposed algorithm can achieve an optimal sum throughput close to the theoretical upper bound, where the performance is significantly improved compared to the scheme based on base station cooperation.