Real-Time LSM-Trees for HTAP Workloads
Authors:
- 1 April 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1208-1220
Abstract
Real-time analytics systems employ hybrid data layouts in which data are stored in different formats throughout their lifecycle. Recent data are stored in a row-oriented format to serve OLTP workloads and support high insert rates, while older data are transformed to a column-oriented format for OLAP access patterns. We observe that a Log-Structured Merge (LSM) Tree is a natural fit for a lifecycle-aware storage engine due to its high write throughput and level-oriented structure, in which records propagate from one level to the next over time. To build a lifecycle-aware storage engine using an LSM-Tree, we make a crucial modification to allow different data layouts in different levels, ranging from purely row-oriented to purely column-oriented, leading to a Real-Time LSM-Tree. We give a cost model and an algorithm to design a Real-Time LSM-Tree that is suitable for a given workload, followed by an experimental evaluation of LASER - a prototype implementation of our idea built on top of the RocksDB key-value store.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Monkey2017
- Hybrid Transactional/Analytical Processing2017
- Bridging the Archipelago between Row-Stores and Column-Stores for Hybrid Workloads2016
- Data Blocks2016
- H2O2014
- The vertica analytic databaseProceedings of the VLDB Endowment, 2012
- HYRISEProceedings of the VLDB Endowment, 2010
- A Case for Fractured Mirrors2002
- The LHAM log-structured history data access methodThe VLDB Journal, 2000
- The log-structured merge-tree (LSM-tree)Acta Informatica, 1996