SpikeVideoFormer: An Efficient Spike-Driven Video Transformer with Hamming Attention and $\mathcal{O}(T)$ Complexity
Preprint
Abstract
Spiking Neural Networks (SNNs) have shown competitive performance to Artificial Neural Networks (ANNs) in various vision tasks, while offering superior energy efficiency. However, existing SNN-based Transformers primarily focus on single-image tasks, emphasizing spatial features while not effectively leveraging SNNs' efficiency in video-based vision tasks. In this paper, we introduce SpikeVideoFormer, an efficient spike-driven video Transformer, featuring linear temporal complexity $\mathcal{O}(T)$. Specifically, we design a spike-driven Hamming attention (SDHA) which provides a theoretically guided adaptation from traditional real-valued attention to spike-driven attention. Building on SDHA, we further analyze various spike-driven space-time attention designs and identify an optimal scheme that delivers appealing performance for video tasks, while maintaining only linear temporal complexity. The generalization ability and efficiency of our model are demonstrated across diverse downstream video tasks, including classification, human pose tracking, and semantic segmentation. Empirical results show our method achieves state-of-the-art (SOTA) performance compared to existing SNN approaches, with over 15\% improvement on the latter two tasks. Additionally, it matches the performance of recent ANN-based methods while offering significant efficiency gains, achieving $\times 16$, $\times 10$ and $\times 5$ improvements on the three tasks. https://github.com/JimmyZou/SpikeVideoFormer
Keywords
All Related Versions
This publication has 36 references indexed in Scilit:
- Integer-Valued Training and Spike-Driven Inference Spiking Neural Network for High-Performance and Energy-Efficient Object Detection2024
- Spike-based dynamic computing with asynchronous sensing-computing neuromorphic chipNature Communications, 2024
- Multi-scale full spike pattern for semantic segmentationNeural Networks, 2024
- An integrative, multiscale view on neural theories of consciousnessNeuron, 2024
- The human imagination: the cognitive neuroscience of visual mental imageryNature Reviews Neuroscience, 2019
- Reevaluating the Sensory Account of Visual Working Memory StorageTrends in Cognitive Sciences, 2017
- Expectation (and attention) in visual cognitionTrends in Cognitive Sciences, 2009
- The role of context in object recognitionTrends in Cognitive Sciences, 2007
- How brains beware: neural mechanisms of emotional attentionTrends in Cognitive Sciences, 2005
- Networks of spiking neurons: The third generation of neural network modelsNeural Networks, 1997