Safe and Accelerated Deep Reinforcement Learning-Based O-RAN Slicing: A Hybrid Transfer Learning Approach
- 28 November 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal on Selected Areas in Communications
- Vol. 42 (2) , 310-325
Abstract
The open radio access network (O-RAN) architecture supports intelligent network control algorithms as one of its core capabilities. Data-driven applications incorporate such algorithms to optimize radio access network (RAN) functions via RAN intelligent controllers (RICs). Deep reinforcement learning (DRL) algorithms are among the main approaches adopted in the O-RAN literature to solve dynamic radio resource management problems. However, despite the benefits introduced by the O-RAN RICs, the practical adoption of DRL algorithms in real network deployments falls behind. This is primarily due to the slow convergence and unstable performance exhibited by DRL agents upon deployment and when encountering previously unseen network conditions. In this paper, we address these challenges by proposing transfer learning (TL) as a core component of the training and deployment workflows for the DRL-based closed-loop control of O-RAN functionalities. To this end, we propose and design a hybrid TL-aided approach that leverages the advantages of both policy reuse and distillation TL methods to provide safe and accelerated convergence in DRL-based O-RAN slicing. We conduct a thorough experiment that accommodates multiple services, including real VR gaming traffic to reflect practical scenarios of O-RAN slicing. We also propose and implement policy reuse and distillation-aided DRL and non-TL-aided DRL as three separate baselines. The proposed hybrid approach shows at least: 7.7% and 20.7% improvements in the average initial reward value and the percentage of converged scenarios, and a 64.6% decrease in reward variance while maintaining fast convergence and enhancing the generalizability compared with the baselines.Keywords
All Related Versions
Funding Information
- Natural Sciences and Engineering Research Council of Canada (RGPIN2019-05667, RGPIN-2021-04050)
This publication has 29 references indexed in Scilit:
- Federated Deep Reinforcement Learning for Open RAN Slicing in 6G NetworksIEEE Communications Magazine, 2022
- Federated Deep Reinforcement Learning for Resource Allocation in O-RAN Slicing2022
- RLOps: Development Life-Cycle of Reinforcement Learning Aided Open RANIEEE Access, 2022
- Artificial Intelligence in Open-Radio Access NetworkIEEE Aerospace and Electronic Systems Magazine, 2022
- Transfer Learning for Wireless Networks: A Comprehensive SurveyProceedings of the IEEE, 2022
- Virtual Reality Gaming on the Cloud: A Reality Check2021
- Transfer Learning-Based Accelerated Deep Reinforcement Learning for 5G RAN Slicing2021
- Bayesian Optimization for Radio Resource Management: Open Loop Power ControlIEEE Journal on Selected Areas in Communications, 2021
- Single and Multi-Agent Deep Reinforcement Learning for AI-Enabled Wireless Networks: A TutorialIEEE Communications Surveys & Tutorials, 2021
- Deep Reinforcement Learning for Resource Management in Network SlicingIEEE Access, 2018