Accelerating ML/DL Applications with Hierarchical Caching on Deduplication Storage Clusters
Open Access
- 20 August 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Big Data
- Vol. PP (6) , 1
Abstract
Large scale machine learning (ML) and deep learning (DL) platforms face challenges when integrated with deduplication enabled storage clusters. In the quest to achieve smart and efficient storage utilization, removal of duplicate data introduces bottlenecks since deduplication alters the I/O transaction layout of the storage system. Therefore, it is critical to address such deduplication overhead for acceleration of ML/DL computation in deduplication storage. Existing state of the art ML/DL storage solutions such as Alluxio and AutoCache adopt non deduplication-aware caching mechanisms, which lacks the much needed performance boost when adopted in deduplication enabled ML/DL clusters. In this paper, we introduce Redup, which eliminates the performance drop caused by enabling deduplication in ML/DL storage clusters. At the core, is a Redup Caching Manager (RDCM), composed of a 2-tier deduplication layout-aware caching mechanism. The RDCM provides an abstraction of the underlying deduplication storage layout to ML/DL applications and provisions a decoupled acceleration of object reconstruction during ML/DL read operations. Our Redup evaluation shows negligible performance drop in ML/DL training performances as compared to a cluster without deduplication, whilst significantly outperforming Alluxio and AutoCache in terms of various performance metrics.Keywords
Funding Information
- Institute for Information and Communications Technology Promotion (2021-0-02051)
This publication has 32 references indexed in Scilit:
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- A Lookahead Read Cache: Improving Read Performance for Deduplication Backup StorageJournal of Computer Science and Technology, 2017
- An Improved HITS Algorithm Based on Analysis of Web Page Links and Web Content Similarity2016
- HANDS: A heuristically arranged non-backup in-line deduplication system2013
- Probabilistic deduplication for cluster-based storage systems2012
- Transferring a generic pedestrian detector towards specific scenes2012
- A Scalable Inline Cluster Deduplication Framework for Big Data Protection2012
- Terrestrial Remotely Sensed Imagery in Support of Public Health: New Avenues of Research Using Object-Based Image AnalysisRemote Sensing, 2011
- RADOS2007
- CRUSH: Controlled, Scalable, Decentralized Placement of Replicated Data2006