An Efficient Deep Reinforcement Learning-Based Automatic Cache Replacement Policy in Cloud Block Storage Systems
- 19 October 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computers
- Vol. 73 (1) , 164-177
Abstract
With the popularity of cloud services, cloud block storage (CBS) systems have been widely deployed by cloud providers. Cloud cache plays a vital role in maintaining high and stable performance in cloud block storage systems. In the past few decades, much research has been conducted on the design of cache replacement policies. Prior work frequently relies on manually-engineered heuristics to capture the most common cache access patterns, or predict the reuse distance and try to identify the blocks that are either cache-friendly or cache-averse. Researchers are now applying recent advances in machine learning to guide cache replacement policy, augmenting or replacing traditional heuristics and data structures. However, most existing approaches depend on a certain environment which restricted their application, e.g., some methods only consider the on-chip cache consisting of program counters (PCs). Moreover, those approaches with attractive hit rates are usually unable to deal with modern irregular workloads, due to the limited feature used. In contrast, we propose a cloud cache replacement framework to automatically learn the relationship between the probability distribution of different replacement policies and workload distribution by using deep reinforcement learning. We train an end-to-end cache replacement policy based on the requested address with two efficient and stable cache replacement policies. Furthermore, by using prioritized experience replay and setting parameter constraints, our framework can accelerate the offline training process without affecting the cloud application. We have evaluated our proposed framework by using block-based I/O traces collected from Alibaba Cloud and Tencent Cloud, two of the largest cloud providers in the world, and several open-source traces. Experimental results show that our method not only outperforms several state-of-the-art cache methods in hit rate, but also reduces request latency and data traffic to the backend storage.Keywords
Funding Information
- NSFC (61832020, 61821003, 82090044)
This publication has 21 references indexed in Scilit:
- Learning I/O Access Patterns to Improve Prefetching in SSDs2021
- Designing a Cost-Effective Cache Replacement Policy using Machine Learning2021
- Experiences with ML-Driven Design: A NoC Case Study2020
- Analysis of and Optimization for Write-dominated Hybrid Storage Nodes in Cloud2019
- Applying Deep Learning to the Cache Replacement Problem2019
- An End-to-End Automatic Cloud Database Tuning System Using Deep Reinforcement Learning2019
- Towards Lightweight and Robust Machine Learning for CDN Caching2018
- DLIRS2018
- Back to the Future: Leveraging Belady's Algorithm for Improved Cache Replacement2016
- CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithmsSoftware: Practice and Experience, 2010