Persistence Initialization: a novel adaptation of the Transformer architecture for time series forecasting
Open Access
- 29 August 2023
- journal article
- research article
- Published by Springer Nature in Applied Intelligence
- Vol. 53 (22) , 26781-26796
Abstract
Time series forecasting is an important problem, with many real world applications. Transformer models have been successfully applied to natural language processing tasks, but have received relatively little attention for time series forecasting. Motivated by the differences between classification tasks and forecasting, we propose PI-Transformer, an adaptation of the Transformer architecture designed for time series forecasting, consisting of three parts: First, we propose a novel initialization method called Persistence Initialization, with the goal of increasing training stability of forecasting models by ensuring that the initial outputs of an untrained model are identical to the outputs of a simple baseline model. Second, we use ReZero normalization instead of Layer Normalization, in order to further tackle issues related to training stability. Third, we use Rotary positional encodings to provide a better inductive bias for forecasting. Multiple ablation studies show that the PI-Transformer is more accurate, learns faster, and scales better than regular Transformer models. Finally, PI-Transformer achieves competitive performance on the challenging M4 dataset, both when compared to the current state of the art, and to recently proposed Transformer models for time series forecasting.Keywords
All Related Versions
Funding Information
- Norges Forskningsråd (312062)
This publication has 13 references indexed in Scilit:
- Highly accurate protein structure prediction with AlphaFoldNature, 2021
- The M4 Competition: 100,000 time series and 61 forecasting methodsInternational Journal of Forecasting, 2020
- A hybrid method of exponential smoothing and recurrent neural networks for time series forecastingInternational Journal of Forecasting, 2020
- A brief history of forecasting competitionsInternational Journal of Forecasting, 2020
- The M4 Competition: Results, findings, conclusion and way forwardInternational Journal of Forecasting, 2018
- Residual LSTM: Design of a Deep Recurrent Architecture for Distant Speech Recognition2017
- A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction2017
- Another look at measures of forecast accuracyInternational Journal of Forecasting, 2006
- Reducing the Dimensionality of Data with Neural NetworksScience, 2006
- The M3-Competition: results, conclusions and implicationsInternational Journal of Forecasting, 2000