ReSpike: residual frames-based hybrid spiking neural networks for efficient action recognition

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Spiking Neural Networks (SNNs) have emerged as a compelling, energy-efficient alternative to traditional Artificial Neural Networks (ANNs) for static image tasks such as image classification and segmentation. However, in the more complex video classification domain, SNN-based methods fall considerably short of ANN-based benchmarks due to the challenges in processing dense frame sequences. To bridge this gap, we propose ReSpike , a hybrid framework that synergizes the strengths of ANNs and SNNs to tackle action recognition tasks with high accuracy and low energy cost. By partitioning film clips into RGB image Key Frames , which primarily capture spatial information, and event-like Residual Frames , which emphasize temporal dynamics cues, ReSpike leverages ANN for processing spatial features and SNN for modeling temporal features. In addition, we propose a multi-scale cross-attention mechanism for effective feature fusion. Compared to state-of-the-art SNN baselines, our ReSpike hybrid architecture demonstrates significant performance improvements (e.g. > 30 % absolute accuracy improvement on HMDB-51, UCF-101, and Kinetics-400). Furthermore, ReSpike achieves comparable performance with prior ANN approaches while bringing better accuracy-energy tradeoff. Code is shared at https://github.com/Intelligent-Computing-Lab-Yale/ReSpike .
Funding Information
  • DoE MMICC center SEA-CROGS (Award #DE-SC0023198)
  • National Science Foundation (CAREER Award, Grant #2312366, Grant #2318152)
  • CoCoSys
  • DARPA (Young Faculty Award)

This publication has 38 references indexed in Scilit: