Ying Yang

By integrating the software-defined networking (SDN) architecture with the machine learning-based knowledge plane, knowledge-defined networking (KDN) is revolutionizing established traffic engineering (TE) methodologies. This paper investigates the challenging joint routing and caching problem in KDN-based networks, managing multiple traffic flows to improve long-term quality-of-service (QoS) performance. This challenge is formulated as a computationally expensive non-convex mixed-integer non-linear programming (MINLP) problem, which exceeds the capacity of heuristic methods to achieve near-optimal solutions. To address this issue, we present DRL-JRC, an actor-critic deep reinforcement learning (DRL) algorithm for adaptive joint routing and caching in KDN-based networks. DRL-JRC orchestrates the optimization of multiple QoS metrics, including end-to-end delay, packet loss rate, load balancing index, and hop count. During offline training, DRL-JRC employs proximal policy optimization (PPO) to smooth the policy optimization process. In addition, the learned policy can be seamlessly integrated with conventional caching solutions during online execution. Extensive experiments demonstrate the comprehensive superiority of DRL-JRC over baseline methods in various scenarios. Meanwhile, DRL-JRC consistently outperforms the heuristic baseline under partial policy deployment during execution. Compared to the average performance of the baseline methods, DRL-JRC reduces the end-to-end delay by 51.14% and the packet loss rate by 40.78%.

Multipath routing remains a challenging issue in traffic engineering (TE) as existing solutions are incapable of handling the evolving network dynamics and stringent quality-of-service (QoS) requirements. To address it, multiagent deep reinforcement learning (MADRL) is a promising technique that provides more elaborate multipath routing strategies. However, prevalent MADRL-based solutions still suffer inapplicability as they fail to ensure both scalability and QoS awareness. In this paper, we leverage the emerging hybrid knowledge-defined networking (KDN) architecture, and propose a collaborative MADRL-based multipath routing algorithm. Two novel mechanisms, i.e., parallel agent replication and periodic policy synchronization, are devised for agent design to ensure the practicality of the proposed method. In addition, an efficient communication mechanism is established to facilitate multiagent collaboration by enabling scalable observation and reward exchange. Featuring a multiagent twin-actor-critic (MA-TAC) learning structure and a proximal policy optimization (PPO) -based training process, the proposed algorithm consists of alternately scheduled execution and training phases for practical deployment. We compare the performance of our proposed method with those of several benchmark methods. Extensive simulation results demonstrate that the proposed method achieves significantly better scalability, QoS awareness, and stability than the benchmark methods under various environment settings.