ModelarDB
Open Access
- 1 July 2018
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in Proceedings of the VLDB Endowment
- Vol. 11 (11) , 1688-1701
Abstract
Industrial systems, e.g., wind turbines, generate big amounts of data from reliable sensors with high velocity. As it is unfeasible to store and query such big amounts of data, only simple aggregates are currently stored. However, aggregates remove fluctuations and outliers that can reveal underlying problems and limit the knowledge to be gained from historical data. As a remedy, we present the distributed Time Series Management System (TSMS) ModelarDB that uses models to store sensor data. We thus propose an online, adaptive multi-model compression algorithm that maintains data values within a user-defined error bound (possibly zero). We also propose (i) a database schema to store time series as models, (ii) methods to push-down predicates to a key-value store utilizing this schema, (iii) optimized methods to execute aggregate queries on models, (iv) a method to optimize execution of projections through static code-generation, and (v) dynamic extensibility that allows new models to be used without recompiling the TSMS. Further, we present a general modular distributed TSMS architecture and its implementation, ModelarDB, as a portable library, using Apache Spark for query processing and Apache Cassandra for storage. An experimental evaluation shows that, unlike current systems, ModelarDB hits a sweet spot and offers fast ingestion, good compression, and fast, scalable online aggregate query processing at the same time. This is achieved by dynamically adapting to data sets using multiple models. The system degrades gracefully as more outliers occur and the actual errors are much lower than the bounds.Keywords
This publication has 22 references indexed in Scilit:
- Spark SQL2015
- Piecewise linear approximation of streaming time series data with max-error guarantees2015
- A time-series compression technique and its application to the smart gridThe VLDB Journal, 2014
- Efficient Indexing and Query Processing of Model-View Sensor Data in the CloudBig Data Research, 2014
- Discretized streams2013
- Indexable online time series segmentation with error bound guaranteeWorld Wide Web, 2013
- A Survey of Model-based Sensor Data Acquisition and Management2012
- CassandraACM SIGOPS Operating Systems Review, 2010
- Online piece-wise linear approximation of numerical streams with precision guaranteesProceedings of the VLDB Endowment, 2009
- MauveDB2006