FireFly-T: High-Throughput Sparsity Exploitation for Spiking Transformer Acceleration with Dual-Engine Overlay Architecture
Preprint
Abstract
Spiking transformers are emerging as a promising architecture that combines the energy efficiency of Spiking Neural Networks (SNNs) with the powerful attention mechanisms of transformers. However, existing hardware accelerators lack support for spiking attention, exhibit limited throughput in exploiting fine-grained sparsity, and struggle with scalable parallelism in sparse computation. To address these, we propose FireFly-T, a dual-engine overlay architecture that integrates a sparse engine for activation sparsity and a binary engine for spiking attention. In the sparse engine, we propose a highthroughput sparse decoder that exploits fine-grained sparsity by concurrently extracting multiple non-zero spikes. To complement this, we introduce a scalable load balancing mechanism with weight dispatch and out-of-order execution, eliminating bank conflicts to support scalable multidimensional parallelism. In the binary engine, we leverage the byte-level write capability of SRAMs to efficiently manipulate the 3D dataflows required for spiking attention with minimal resource overhead. We also optimize the core AND-PopCount operation in spiking attention through a LUT6-based implementation, improving timing closure and reducing LUT utilization on Xilinx FPGAs. As an overlay architecture, FireFly-T further incorporates an orchestrator that dynamically manipulates input dataflows with flexible adaptation for diverse network topologies, while ensuring efficient resource utilization and maintaining high throughput. Experimental results demonstrate that our accelerator achieves $1.39\times$ and $2.40\times$ higher energy efficiency, as well as $4.21\times$ and $7.10\times$ greater DSP efficiency, compared to FireFly v2 and the transformer-enabled SpikeTA, respectively. These results highlight its potential as an efficient hardware platform for spiking transformer.
Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Advancing Neuromorphic Architecture Toward Emerging Spiking Neural Network on FPGAIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2025
- An Energy-Efficient Unstructured Sparsity-Aware Deep SNN Accelerator With 3-D Computation ArrayIEEE Journal of Solid-State Circuits, 2024
- FireFly-S: Exploiting Dual-Side Sparsity for Spiking Neural Networks Acceleration With Reconfigurable Spatial ArchitectureIEEE Transactions on Circuits and Systems I: Regular Papers, 2024
- SiBrain: A Sparse Spatio-Temporal Parallel Neuromorphic Architecture for Accelerating Spiking Convolution Neural Networks With Low LatencyIEEE Transactions on Circuits and Systems I: Regular Papers, 2024
- FireFly v2: Advancing Hardware Support for High-Performance Spiking Neural Network With a Spatiotemporal FPGA AcceleratorIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2024
- BrainCog: A spiking neural network based, brain-inspired cognitive intelligence engine for brain-inspired AI and brain simulationPatterns, 2023
- DeepFire2: A Convolutional Spiking Neural Network Accelerator on FPGAsIEEE Transactions on Computers, 2023
- ViA: A Novel Vision-Transformer Accelerator Based on FPGAIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2022
- Networks of spiking neurons: The third generation of neural network modelsNeural Networks, 1997
- A Suggestion for a Fast MultiplierIEEE Transactions on Electronic Computers, 1964