SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot
Authors:
Preprint
Abstract
State-space language models such as Mamba match Transformer quality while permitting linear complexity inference, yet still comprise billions of parameters that hinder deployment. Existing one-shot pruning methods are tailored to attention blocks and fail to account for the time-shared and discretized state-transition matrix at the heart of the selective state-space module (SSM). In this paper, we introduce SparseSSM, the first training-free pruning framework that extends the classic optimal brain surgeon (OBS) framework to state space architectures. Our layer-wise algorithm (i) derives an approximate second-order saliency score that aggregates Hessian-trace information across time steps, (ii) incorporates a component sensitivity analysis to guide feed-forward network (FFN) pruning, which also sheds light on where redundancy resides in mamba architecture, (iii) can be easily extended to semi-structured and structured sparsity. Empirically, we prune 50% of SSM weights without fine-tuning and observe no zero-shot accuracy loss, achieving the current state-of-the-art pruning algorithm for Mamba-based LLMs.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- On Pruning State-Space LLMs2025
- Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models2025
- One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Models2024
- Structured Optimal Brain Pruning for Large Language Models2024
- LLM-Pruner: On the Structural Pruning of Large Language Models2023
- Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning2022
- PIQA: Reasoning about Physical Commonsense in Natural LanguageProceedings of the AAAI Conference on Artificial Intelligence, 2020
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering2018
- Channel Pruning for Accelerating Very Deep Neural Networks2017