On Pruning State-Space LLMs

Authors:
  • TGTamer Ghattas
    Tamer Ghattas
  • MHMichael Hassid
    Michael Hassid
  • RSRoy Schwartz
    Roy Schwartz

Abstract
Recent work proposed state-space models (SSMs) as an efficient alternative to transformer-based LLMs. Can these models be pruned to further reduce their computation costs? We adapt several pruning methods to the SSM structure, and apply them to four SSM-based LLMs across multiple tasks. We find that such models are quite robust to some pruning methods (e.g. WANDA), while using other methods lead to fast performance degradation.