Compressed Data Direct Computing for Databases
- 18 September 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Knowledge and Data Engineering
- Vol. 36 (5) , 1902-1918
Abstract
Directly performing operations on compressed data has been proven to be a big success facing big data problems in modern data management systems. These systems have demonstrated significant compression benefits and performance improvement for data analytics applications. However, current systems only focus on data queries, while a complete big data system must support both data query and data manipulation. To solve this problem, we develop CompressDB, which is a new storage engine that can support data processing for databases without decompression. CompressDB has the following advantages. First, CompressDB utilizes context-free grammar to compress data, and supports both data query and data manipulation. Second, for adaptability, we integrate CompressDB to file systems so that a wide range of databases can directly use CompressDB without any change. Third, we enable operation pushdown to storage so that we can perform data query and manipulation in storage systems without bringing large data to memory for high efficiency. We validate the efficacy of CompressDB supporting various kinds of database systems, including SQLite, MySQL, LevelDB, MongoDB, ClickHouse, and Neo4j. We evaluate our method using seven real-world datasets with various lengths, structures, and content in both single node and cluster environments. Experiments show that CompressDB achieves 40% throughput improvement and 44% latency reduction, along with 1.75 compression ratio on average.Keywords
Funding Information
- National Natural Science Foundation of China (62322213, 62172419)
- Beijing Nova
- Public Computing Cloud, Renmin University of China
- building world-class universities
- Renmin University of China
This publication has 70 references indexed in Scilit:
- An Adaptive Elastic Multi-model Big Data Analysis and Information Extraction SystemData Science and Engineering, 2022
- Efficient trajectory compression and range query processingWorld Wide Web, 2022
- TADOC: Text analytics directly on compressionThe VLDB Journal, 2020
- GraCT: A Grammar-based Compressed Index for Trajectory DataInformation Sciences, 2019
- Finger Search in Grammar-Compressed StringsTheory of Computing Systems, 2018
- Compact Data Structures2016
- A spatiotemporal compression based approach for efficient big data processing on CloudJournal of Computer and System Sciences, 2014
- Comprehensive analysis of big data variety landscapeInternational Journal of Parallel, Emergent and Distributed Systems, 2014
- Compressed Suffix Trees with Full FunctionalityTheory of Computing Systems, 2007
- New text indexing functionalities of the compressed suffix arraysJournal of Algorithms, 2003