A Multiple Compression Approach using Attribute-based Signatures
Open Access
- 10 February 2025
- journal article
- Published by CERN in Open Research Europe
- Vol. 5, 49
Abstract
Background With the increasing volume of data collected for advanced analytical and AI applications, data storage remains a significant challenge. Despite advancements in storage technologies, the cost of maintaining vast datasets continues to grow. Compression techniques have been widely used to address this issue, but existing systems primarily rely on a single, typically lossless method, which limits adaptability to varying data characteristics. Methods This paper introduces COMPASS, a multiple compression approach that applies different compression techniques to different subsets of data within a database. COMPASS partitions relational data into rows or columns and selects the most suitable compression scheme for individual columns or column groups. Two versions of COMPASS are proposed: (i) COMPASS-D, which utilizes K-Means clustering based on data values; and (ii) COMPASS-E, which employs K-Means clustering based on column entropy to group similar columns efficiently. The effectiveness of COMPASS is evaluated using the Envmon dataset, a real-world environmental monitoring database, and compared against monolithic compression methods. Results Experimental results demonstrate that COMPASS significantly reduces disk space usage compared to traditional compression techniques. COMPASS-E achieves superior performance in terms of compression time and proximity to the optimal compression ratio, outperforming COMPASS-D. In worst-case scenarios, COMPASS methods offer 22% more savings compared to baseline techniques, with best-case savings reaching 56% (~2× improvement). Conclusion The proposed COMPASS framework offers a flexible and adaptive approach to database compression by leveraging multiple schemes tailored to different data subsets. This results in improved storage efficiency and reduced computational overhead. Future work will explore additional data characteristics and clustering methods to further enhance COMPASS’s adaptability and efficiency.Keywords
Funding Information
- European Commission
- Research and Innovation Foundation (CONCEPT/0823/0002)
This publication has 27 references indexed in Scilit:
- Towards a Signature Based Compression Technique for Big Data Storage2023
- Data management in training AI chatbot with personality based on granular computing2022
- Analysis of Dimensionality Reduction Techniques on Big DataIEEE Access, 2020
- Comparison of Lossless Data Compression Techniques2020
- Continuous decaying of telco big data with data postdictionGeoInformatica, 2019
- Parallel Implementation of Lossy Data Compression for Temporal Data Sets2016
- Data Reduction2016
- HaoLap: A Hadoop based OLAP system for big dataJournal of Systems and Software, 2015
- Silhouettes: A graphical aid to the interpretation and validation of cluster analysisJournal of Computational and Applied Mathematics, 1987
- A Mathematical Theory of CommunicationBell System Technical Journal, 1948