MS-SSM: A Multi-Scale State Space Model for Efficient Sequence Modeling
Preprint
Abstract
State-space models (SSMs) have recently attention as an efficient alternative to computationally expensive attention-based models for sequence modeling. They rely on linear recurrences to integrate information over time, enabling fast inference, parallelizable training, and control over recurrence stability. However, traditional SSMs often suffer from limited effective memory, requiring larger state sizes for improved recall. Moreover, existing SSMs struggle to capture multi-scale dependencies, which are essential for modeling complex structures in time series, images, and natural language. This paper introduces a multi-scale SSM framework that addresses these limitations by representing sequence dynamics across multiple resolution and processing each resolution with specialized state-space dynamics. By capturing both fine-grained, high-frequency patterns and coarse, global trends, MS-SSM enhances memory efficiency and long-range modeling. We further introduce an input-dependent scale-mixer, enabling dynamic information fusion across resolutions. The proposed approach significantly improves sequence modeling, particularly in long-range and hierarchical tasks, while maintaining computational efficiency. Extensive experiments on benchmarks, including Long Range Arena, hierarchical reasoning, time series classification, and image recognition, demonstrate that MS-SSM consistently outperforms prior SSM-based models, highlighting the benefits of multi-resolution processing in state-space architectures.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- Evidence of a predictive coding hierarchy in the human brain listening to speechNature Human Behaviour, 2023
- Classification of long sequential data using circular dilated convolutional neural networksNeurocomputing, 2022
- Stationary wavelet transform based ECG signal denoising methodISA Transactions, 2021
- H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences2021
- Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XLIEEE Journal of Biomedical and Health Informatics, 2020
- Evidence for a hierarchy of predictions and prediction errors in human cortexProceedings of the National Academy of Sciences, 2011
- IMAGE Resolution Enhancement by Using Discrete and Stationary Wavelet DecompositionIEEE Transactions on Image Processing, 2010
- FEATURE EXTRACTION OF BRAIN MRI BY STATIONARY WAVELET TRANSFORM AND ITS APPLICATIONSJournal of Biological Systems, 2010
- An extended class of scale-invariant and recursive scale space filtersIEEE Transactions on Pattern Analysis and Machine Intelligence, 1995
- Ten Lectures on WaveletsThe Journal of the Acoustical Society of America, 1993