Consistently faster and smaller compressed bitmaps with Roaring
Authors:
Preprint
Abstract
Compressed bitmap indexes are used in databases and search engines. Many bitmap compression techniques have been proposed, almost all relying primarily on run-length encoding (RLE). However, on unsorted data, we can get superior performance with a hybrid compression technique that uses both uncompressed bitmaps and packed arrays inside a two-level tree. An instance of this technique, Roaring, has recently been proposed. Due to its good performance, it has been adopted by several production platforms (e.g., Apache Lucene, Apache Spark, Apache Kylin and Druid). Yet there are cases where run-length encoded bitmaps are smaller than the original Roaring bitmaps---typically when the data is sorted so that the bitmaps contain long compressible runs. To better handle these cases, we build a new Roaring hybrid that combines uncompressed bitmaps, packed arrays and RLE compressed segments. The result is a new Roaring format that compresses better. Overall, our new implementation of Roaring can be several times faster (up to two orders of magnitude) than the implementations of traditional RLE-based alternatives (WAH, Concise, EWAH) while compressing better. We review the design choices and optimizations that make these good results possible.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Hybrid query optimization for hard-to-compress bit-vectorsThe VLDB Journal, 2015
- Better bitmap performance with Roaring bitmapsSoftware: Practice and Experience, 2015
- SIMD compression and the intersection of sorted integersSoftware: Practice and Experience, 2015
- A survey of bitmap index compression algorithms for Big DataTsinghua Science and Technology, 2015
- Compressed bitmap indexes: beyond unions and intersectionsSoftware: Practice and Experience, 2014
- A tunable compression framework for bitmap indices2014
- Decoding billions of integers per second through vectorizationSoftware: Practice and Experience, 2013
- Concise: Compressed ‘n’ Composable Integer SetInformation Processing Letters, 2010
- Sorting improves word-aligned bitmap indexesData & Knowledge Engineering, 2010
- Breaking the Curse of Cardinality on Bitmap Indexes2008