Preformer: Predictive Transformer with Multi-Scale Segment-Wise Correlations for Long-Term Time Series Forecasting
Authors:
- 4 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-5
Abstract
In long-term time series forecasting, most Transformer-based methods adopt the standard point-wise attention mechanism, which not only has high complexity but also cannot explicitly capture the predictive dependencies from contexts since the corresponding key and value are transformed from the same point. This paper proposes a predictive Transformer-based model called Preformer. Preformer introduces a novel efficient Multi-Scale Segment-Correlation mechanism that divides time series into segments and utilizes segment-wise correlation-based attention to replace point-wise attention. A multi-scale structure is developed to aggregate dependencies at different temporal scales and facilitate the selection of segment length. Preformer further designs a predictive paradigm for decoding, where the key and value come from two successive segments rather than the same segment. Experiments demonstrate that Preformer outperforms other Transformer-based models. The codes are available at https://github.com/ddz16/Preformer.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China
- Renmin University of China
This publication has 6 references indexed in Scilit:
- Temporal Fusion Transformers for interpretable multi-horizon time series forecastingInternational Journal of Forecasting, 2021
- Informer: Beyond Efficient Transformer for Long Sequence Time-Series ForecastingProceedings of the AAAI Conference on Artificial Intelligence, 2021
- DeepAR: Probabilistic forecasting with autoregressive recurrent networksInternational Journal of Forecasting, 2020
- Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks2018
- Attend and Diagnose: Clinical Time Series Analysis Using Attention ModelsProceedings of the AAAI Conference on Artificial Intelligence, 2018
- A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction2017