RITA: Group Attention is All You Need for Timeseries Analytics
Preprint
Abstract
Timeseries analytics is of great importance in many real-world applications. Recently, the Transformer model, popular in natural language processing, has been leveraged to learn high quality feature embeddings from timeseries, core to the performance of various timeseries analytics tasks. However, the quadratic time and space complexities limit Transformers' scalability, especially for long timeseries. To address these issues, we develop a timeseries analytics tool, RITA, which uses a novel attention mechanism, named group attention, to address this scalability issue. Group attention dynamically clusters the objects based on their similarity into a small number of groups and approximately computes the attention at the coarse group granularity. It thus significantly reduces the time and space complexity, yet provides a theoretical guarantee on the quality of the computed attention. The dynamic scheduler of RITA continuously adapts the number of groups and the batch size in the training process, ensuring group attention always uses the fewest groups needed to meet the approximation quality requirement. Extensive experiments on various timeseries datasets and analytics tasks demonstrate that RITA outperforms the state-of-the-art in accuracy and is significantly faster -- with speedups of up to 63X.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- InceptionTime: Finding AlexNet for time series classificationData Mining and Knowledge Discovery, 2020
- ROCKET: exceptionally fast and accurate time series classification using random convolutional kernelsData Mining and Knowledge Discovery, 2020
- DeepAR: Probabilistic forecasting with autoregressive recurrent networksInternational Journal of Forecasting, 2020
- TS-CHIEF: a scalable and accurate forest algorithm for time series classificationData Mining and Knowledge Discovery, 2020
- SciPy 1.0: fundamental algorithms for scientific computing in PythonNature Methods, 2020
- Anomaly Detection for IoT Time-Series Data: A SurveyIEEE Internet of Things Journal, 2019
- Billion-Scale Similarity Search with GPUsIEEE Transactions on Big Data, 2019
- Deep learning for time series classification: a reviewData Mining and Knowledge Discovery, 2019
- Dimensionality Reduction for Fast Similarity Search in Large Time Series DatabasesKnowledge and Information Systems, 2001
- Least squares quantization in PCMIEEE Transactions on Information Theory, 1982