ByteSeries
Authors:
Open Access
- 12 October 2020
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 60-73
Abstract
Monitoring large-scale and complex systems often generates high-dimensional and highly dynamic time series data. In such a scenario, massive metadata has to be maintained to support efficient querying, whose large footprint poses great challenges to in-memory databases. In this paper, we present ByteSeries, an in-memory time series database that is designed specifically for large-scale monitoring systems to manage high-dimensional time series. We start with an analysis of the production data and workload at ByteDance's metric monitoring system, which contains over 10 billion time series dimensions. The observation of high overhead of metadata management in high-dimensional time series data calls for a rethink of time series database systems. Byte-Series's memory structure employs the novel Compressed Inverted Index to effectively compress metadata while maintaining high efficiency for multi-dimensional queries. In addition, an algorithm is proposed to effectively convert data into compressed form without sacrificing the data ingestion throughput. We experimentally evaluate ByteSeries by comparing it with ByteDance's original production system, tsdc, as well as two open-source systems, namely Gorilla and Prometheus. We show that ByteSeries significantly improves over ByteDance's original production system by 1) reducing the memory footprint of metadata by 60% and the whole memory consumption by 50%, and 2) speeding up multi-dimensional queries by 1.8x-10.7x.Keywords
This publication has 9 references indexed in Scilit:
- Time Series Management Systems: A SurveyIEEE Transactions on Knowledge and Data Engineering, 2017
- GorillaProceedings of the VLDB Endowment, 2015
- Dynamic Time Warping Averaging of Time Series Allows Faster and More Accurate Classification2014
- Time Series Classification under More Realistic Assumptions2013
- Searching and mining trillions of time series subsequences under dynamic time warping2012
- Visually mining and monitoring massive time series2004
- Finding surprising patterns in a time series database in linear time and space2002
- Locally adaptive dimensionality reduction for indexing large time series databases2001
- The log-structured merge-tree (LSM-tree)Acta Informatica, 1996