Multi-Agent Reinforcement Learning for Cooperative Coded Caching via Homotopy Optimization
- 23 March 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Wireless Communications
- Vol. 20 (8) , 5258-5272
Abstract
Introducing cooperative coded caching into small cell networks is a promising approach to reducing traffic loads. By encoding content via maximum distance separable (MDS) codes, coded fragments can be collectively cached at small-cell base stations (SBSs) to enhance caching efficiency. However, content popularity is usually time-varying and unknown in practice. As a result, cached content is anticipated to be intelligently updated by taking into account limited caching storage and interactive impacts among SBSs. In response to these challenges, we propose a multi-agent deep reinforcement learning (DRL) framework to intelligently update cached content in dynamic environments. With the goal of minimizing long-term expected fronthaul traffic loads, we first model dynamic coded caching as a cooperative multi-agent Markov decision process. Owing to the use of MDS coding, the resulting decision-making falls into a class of constrained reinforcement learning problems with continuous decision variables. To deal with this difficulty, we custom-build a novel DRL algorithm by embedding homotopy optimization into a deep deterministic policy gradient formalism. Next, to empower the caching framework with an effective trade-off between complexity and performance, we propose centralized, and partially and fully decentralized caching controls by applying the derived DRL approach. Simulation results demonstrate the superior performance of the proposed multi-agent framework.Keywords
All Related Versions
Funding Information
- U.S. National Science Foundation (CCF-1908308)
- EU Marie Sklodowska-Curie Actions Project entitled “High-reliability Low-latency Communications with Network Coding”
- Swedish Foundation for International Cooperation in Research and Higher Education (STINT), project “Efficient and Secure Distributed Machine Learning with Gradient Descend”
- National Natural Science Foundation of China (61872184)
This publication has 32 references indexed in Scilit:
- A Learning-Based Approach to Caching in Heterogenous Small Cell NetworksIEEE Transactions on Communications, 2016
- Optimizing MDS Codes for Caching at the Edge2015
- Distributed Caching for Data Dissemination in the Downlink of Heterogeneous NetworksIEEE Transactions on Communications, 2015
- Delay performance analysis of cooperative cell caching in future mobile networks2015
- Deep learningNature, 2015
- Human-level control through deep reinforcement learningNature, 2015
- Living on the edge: The role of proactive caching in 5G wireless networksIEEE Communications Magazine, 2014
- Fundamental Limits of CachingIEEE Transactions on Information Theory, 2014
- On the convergence of inexact block coordinate descent methods for constrained optimizationEuropean Journal of Operational Research, 2013
- Mobility modeling in wireless networksACM SIGMOBILE Mobile Computing and Communications Review, 2001