SampDedup: Sampling Prediction for Efficient Inline Data Deduplication on Non-volatile Memory
Open Access
- 23 July 2025
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Architecture and Code Optimization
Abstract
Data deduplication is an effective technique for reducing redundant data storage space in various storage systems. Generally, deduplication consists of four steps: chunking, fingerprinting, fingerprint lookup, and data management. Recently, Non-volatile Memory (NVM) as an emerging storage device has received widespread attention. Directly applying the deduplication technique on NVM for storage cost savings faces many challenges: a) deduplication on NVM devices suffers from computation bottleneck instead of the I/O bottleneck faced by deduplication on traditional storage devices (such as HDD and SSD); b) new fingerprint indexes and metadata are required to be re-designed to adapt to NVM characteristics; c) inline deduplication on NVM is more sensitive to the latency. To solve these challenges, we propose a novel Samp ling prediction-based inline data Dedup lication method (
SampDedup
) on NVM devices. It aims to ensure high deduplication ratios while reducing computation costs and latency by optimizing data chunking , fingerprinting , and fingerprint lookup . a) For data chunking , a sampling prediction-based chunking method ( SampChunk ) is proposed to leverage chunk similarity to distinguish duplicate chunks and skip them for chunking. This method can be easily integrated into most sliding-window based and non-window based CDC chunking algorithms. b) For fingerprinting , the commonly used SHA-1 algorithm is further optimized to reduce the extra computational overhead introduced by SampChunk, and an asynchronous fingerprinting method is proposed to reduce the fingerprinting latency of unique chunks. c) For fingerprint lookup , we design a header fingerprint index and metadata table for each data chunk constructed by SampChunk on NVM, and we use a fast-read buffer to replace the traditional slow LRU cache to improve search efficiency. Experiments on 4 real-world datasets demonstrate that SampDedup consistently presents high inline data deduplication ratios on NVM with different workloads and data partitioning algorithms while saving more than \(90\% \) chunking time compared with state-of-the-art deduplication baselines.
Keywords
This publication has 10 references indexed in Scilit:
- Optimizing the Performance of Consistency-Aware Deduplication Using Persistent MemoryIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2023
- Low‐overhead inline deduplication for persistent memoryTransactions on Emerging Telecommunications Technologies, 2020
- The Design of Fast Content-Defined Chunking for Data Deduplication Based Storage SystemsIEEE Transactions on Parallel and Distributed Systems, 2020
- Sketching Volume Capacities in Deduplicated StorageACM Transactions on Storage, 2019
- NV-Dedup: High-Performance Inline Deduplication for Non-Volatile MemoryIEEE Transactions on Computers, 2017
- A Comprehensive Study of the Past, Present, and Future of Data DeduplicationProceedings of the IEEE, 2016
- Ddelta: A deduplication-inspired fast delta compression approachPerformance Evaluation, 2014
- WAN-optimized replication of backup datasets using stream-informed delta compressionACM Transactions on Storage, 2012
- A study of practical deduplicationACM Transactions on Storage, 2012
- Cuckoo hashingJournal of Algorithms, 2004