Offline Reinforcement Learning for Wireless Network Optimization With Mixture Datasets
- 10 May 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Wireless Communications
- Vol. 23 (10) , 12703-12716
Abstract
The recent development of reinforcement learning (RL) has boosted the adoption of online RL for wireless radio resource management (RRM). However, online RL algorithms require direct interactions with the environment, which may be undesirable given the potential performance loss due to the unavoidable exploration in RL. In this work, we first explore the use of offline RL algorithms in solving the RRM problem. We evaluate several state-of-the-art offline RL algorithms for a practical RRM problem that aims at maximizing a linear combination of total rates and 5-percentile rates via user scheduling. Our findings indicate that the performance of offline RL for the RRM problem is heavily contingent upon the behavior policy deployed for data collection. We propose an innovative offline RL approach utilizing heterogeneous datasets from various behavior policies. This method demonstrates that a strategic mixture of datasets enables near-optimal RL policy generation, even with suboptimal behavior policies. Additionally, we introduce two enhancements: an ensemble-based policy to augment dataset mixture training efficiency, and a novel offline-to-online strategy for seamless adaptation to new environments. Our data mixture approach achieves over 95% efficiency of an online RL agent in the absence of expert data. The ensemble algorithm notably reduces training duration by half compared to the data mixture method. Furthermore, our model, when applied with offline-to-online fine-tuning, surpasses existing benchmarks by approximately 5% in our user scheduling problem.Keywords
All Related Versions
Funding Information
- U.S. National Science Foundation (CNS-2002902, CNS-2003131, ECCS-2029978, ECCS-2030026, ECCS-2143559, SII-2132700)
- U.S. NSF (CNS-1956276, CNS-2003131, CNS-2030026)
This publication has 19 references indexed in Scilit:
- Offline Reinforcement Learning for Wireless Network Optimization with Mixture Datasets2023
- Distributed MARL for Scheduling in Conflict Graphs2023
- Multi-Agent Reinforcement Learning for Wireless User Scheduling: Performance, Scalablility, and Generalization2022
- Deep Reinforcement Learning for Joint Spectrum and Power Allocation in Cellular Networks2021
- Deep Actor-Critic Learning for Distributed Power Control in Wireless Mobile Networks2020
- Deep Reinforcement Learning based Wireless Network Optimization: A Comparative Study2020
- Joint Power Control and Channel Allocation for Interference Mitigation Based on Reinforcement LearningIEEE Access, 2019
- Multi-Agent Deep Reinforcement Learning for Dynamic Power Allocation in Wireless NetworksIEEE Journal on Selected Areas in Communications, 2019
- Power Allocation in Multi-User Cellular Networks with Deep Q Learning Approach2019
- ITLinQ: A New Approach for Spectrum Sharing in Device-to-Device Communication SystemsIEEE Journal on Selected Areas in Communications, 2014