MR-Transformer: Multiresolution Transformer for Multivariate Time Series Prediction
- 6 November 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 36 (1) , 1171-1183
Abstract
Multivariate time series (MTS) prediction has been studied broadly, which is widely applied in real-world applications. Recently, transformer-based methods have shown the potential in this task for their strong sequence modeling ability. Despite progress, these methods pay little attention to extracting short-term information in the context, while short-term patterns play an essential role in reflecting local temporal dynamics. Moreover, we argue that there are both consistent and specific characteristics among multiple variables, which should be fully considered for MTS modeling. To this end, we propose a multiresolution transformer (MR-Transformer) for MTS prediction, modeling MTS from both the temporal and the variable resolution. Specifically, for the temporal resolution, we design a long short-term transformer. We first split the sequence into nonoverlapping segments in an adaptive way and then extract short-term patterns within segments, while long-term patterns are captured by the inherent attention mechanism. Both of them are aggregated together to capture the temporal dependencies. For the variable resolution, besides the variable-consistent features learned by long short-term transformer, we also design a temporal convolution module to capture the specific features of each variable individually. MR-Transformer enhances the MTS modeling ability by combining multiresolution features between both time steps and variables. Extensive experiments conducted on real-world time series datasets show that MR-Transformer significantly outperforms the state-of-the-art MTS prediction models. The visualization analysis also demonstrates the effectiveness of the proposed model.Keywords
Funding Information
- National Natural Science Foundation of China (62272173)
- National Natural Science Foundation of China (61872148)
- Natural Science Foundation of Guangdong Province (2022A1515010179)
- Natural Science Foundation of Guangdong Province (2019A1515010768)
- Science and Technology Planning Project of Guangdong Province (2023A0505050106)
This publication has 39 references indexed in Scilit:
- DeepAR: Probabilistic forecasting with autoregressive recurrent networksInternational Journal of Forecasting, 2020
- Adaptive Temporal-Frequency Network for Time-Series ForecastingIEEE Transactions on Knowledge and Data Engineering, 2020
- A Comprehensive Survey on Graph Neural NetworksIEEE Transactions on Neural Networks and Learning Systems, 2020
- The M4 Competition: 100,000 time series and 61 forecasting methodsInternational Journal of Forecasting, 2020
- DSANet2019
- Multi-Horizon Time Series Forecasting with Temporal Attention Learning2019
- Temporal pattern attention for multivariate time series forecastingMachine Learning, 2019
- Temporal Attention-Augmented Bilinear Network for Financial Time-Series Data AnalysisIEEE Transactions on Neural Networks and Learning Systems, 2018
- Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks2018
- Another look at measures of forecast accuracyInternational Journal of Forecasting, 2006