SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiation
Open Access
- 24 March 2024
- journal article
- research article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 38 (10) , 10998-11006
Abstract
Large language Models (LLMs), though growing exceedingly powerful, comprises of orders of magnitude less neurons and synapses than the human brain. However, it requires significantly more power/energy to operate. In this work, we propose a novel bio-inspired spiking language model (LM) which aims to reduce the computational cost of conventional LMs by drawing motivation from the synaptic information flow in the brain. In this paper, we demonstrate a framework that leverages the average spiking rate of neurons at equilibrium to train a neuromorphic spiking LM using implicit differentiation technique, thereby overcoming the non-differentiability problem of spiking neural network (SNN) based algorithms without using any type of surrogate gradient. The steady-state convergence of the spiking neurons also allows us to design a spiking attention mechanism, which is critical in developing a scalable spiking LM. Moreover, the convergence of average spiking rate of neurons at equilibrium is utilized to develop a novel ANN-SNN knowledge distillation based technique wherein we use a pre-trained BERT model as “teacher” to train our “student” spiking architecture. While the primary architecture proposed in this paper is motivated by BERT, the technique can be potentially extended to different kinds of LLMs. Our work is the first one to demonstrate the performance of an operational spiking LM architecture on multiple different tasks in the GLUE benchmark. Our implementation source code is available at https://github.com/NeuroCompLab-psu/SpikingBERT.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- LaSNN: Layer-wise ANN-to-SNN distillation for effective and efficient training in deep spiking neural networksNeurocomputing, 2025
- Sequence Learning Using Equilibrium Propagation2023
- The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models2022
- Advancing Neuromorphic Computing With Loihi: A Survey of Results and OutlookProceedings of the IEEE, 2021
- TinyBERT: Distilling BERT for Natural Language Understanding2020
- Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural NetworksIEEE Signal Processing Magazine, 2019
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding2018
- Equilibrium Propagation: Bridging the Gap between Energy-Based Models and BackpropagationFrontiers in Computational Neuroscience, 2017
- Converting Static Image Datasets to Spiking Neuromorphic Datasets Using SaccadesFrontiers in Neuroscience, 2015