SCFormer: Structured Channel-wise Transformer with Cumulative Historical State for Multivariate Time Series Forecasting
Preprint
Abstract
The Transformer model has shown strong performance in multivariate time series forecasting by leveraging channel-wise self-attention. However, this approach lacks temporal constraints when computing temporal features and does not utilize cumulative historical series effectively.To address these limitations, we propose the Structured Channel-wise Transformer with Cumulative Historical state (SCFormer). SCFormer introduces temporal constraints to all linear transformations, including the query, key, and value matrices, as well as the fully connected layers within the Transformer. Additionally, SCFormer employs High-order Polynomial Projection Operators (HiPPO) to deal with cumulative historical time series, allowing the model to incorporate information beyond the look-back window during prediction. Extensive experiments on multiple real-world datasets demonstrate that SCFormer significantly outperforms mainstream baselines, highlighting its effectiveness in enhancing time series forecasting. The code is publicly available at https://github.com/ShiweiGuo1995/SCFormerKeywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Transformer: A General Framework from Machine Translation to OthersMachine Intelligence Research, 2023
- Long sequence time-series forecasting with deep learning: A surveyInformation Fusion, 2023
- Vision Transformers in medical computer vision—A contemplative retrospectionEngineering Applications of Artificial Intelligence, 2023
- Applications of transformer-based language models in bioinformatics: a surveyBioinformatics Advances, 2023
- Memory-based Transformer with shorter window and longer horizon for multivariate time series forecastingPattern Recognition Letters, 2022
- SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction2022
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness2022
- State-of-the-art augmented NLP transformer models for direct and single-step retrosynthesisNature Communications, 2020
- Deep Learning with Long Short-Term Memory for Time Series PredictionIEEE Communications Magazine, 2019