Towards a Signature Based Compression Technique for Big Data Storage
Authors:
- 1 April 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 100-104
Abstract
Although the volume of stored data doubles every year, storage capacity costs decline only at a rate of less than 1/5 per year. At the same time, data is stored in multiple physical locations and remotely retrieved from multiple sites. Thus, minimizing data storage costs while maintaining data fidelity and efficient retrieval is still a key challenge in database systems. In addition to the raw big data, its associated metadata and indexes equally demand tremendous storage that impacts the I/O footprint of data centers. In this vision paper, we propose a new signature-based compression (SIBACO) technique that is able to: (i) incrementally store big data in an efficient way; and (ii) improve the retrieval time for data-intensive applications. SIBACO achieves higher compression ratios by combining and compressing columns differently based on the type and distribution of data and can be easily integrated with column and hybrid stores. We evaluate our proposed tool using real datasets showing that SIBACO outperforms "monolithic" compression schemes in terms of storage cost.Keywords
Funding Information
- Research Promotion Foundation
This publication has 19 references indexed in Scilit:
- Decaying Telco Big Data with Data Postdiction2018
- Towards Real-Time Road Traffic Analytics using Telco Big Data2017
- SPATE: Compacting and Exploring Telco Big Data2017
- Efficient Exploration of Telco Big Data with Compression and Decaying2017
- City-Scale Localization with Telco Big Data2016
- Telco Churn Prediction with Big Data2015
- Compression aware physical database designProceedings of the VLDB Endowment, 2011
- Self-organizing tuple reconstruction in column-stores2009
- Integrating compression and execution in column-oriented database systems2006
- A Mathematical Theory of CommunicationBell System Technical Journal, 1948