Language Modeling on a SpiNNaker 2 Neuromorphic Chip
Preprint
Abstract
As large language models continue to scale in size rapidly, so too does the computational power required to run them. Event-based networks on neuromorphic devices offer a potential way to reduce energy consumption for inference significantly. However, to date, most event-based networks that can run on neuromorphic hardware, including spiking neural networks (SNNs), have not achieved task performance even on par with LSTM models for language modeling. As a result, language modeling on neuromorphic devices has seemed a distant prospect. In this work, we demonstrate the first-ever implementation of a language model on a neuromorphic device - specifically the SpiNNaker 2 chip - based on a recently published event-based architecture called the EGRU. SpiNNaker 2 is a many-core neuromorphic chip designed for large-scale asynchronous processing, while the EGRU is architected to leverage such hardware efficiently while maintaining competitive task performance. This implementation marks the first time a neuromorphic language model matches LSTMs, setting the stage for taking task performance to the level of large language models. We also demonstrate results on a gesture recognition task based on inputs from a DVS camera. Overall, our results showcase the feasibility of this neuro-inspired neural network in hardware, highlighting significant gains versus conventional hardware in energy efficiency for the common use case of single batch inference.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit DifferentiationProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Cerebras Architecture Deep Dive: First Look Inside the Hardware/Software Co-Design for Deep LearningIEEE Micro, 2023
- A 95.6-TOPS/W Deep Learning Inference Accelerator With Per-Vector Scaled 4-bit Quantization in 5 nmIEEE Journal of Solid-State Circuits, 2023
- A Long Short-Term Memory for AI Applications in Spike-based Neuromorphic HardwareNature Machine Intelligence, 2022
- A Low Power, Fully Event-Based Gesture Recognition System2017
- A 128 X 128 120db 30mw asynchronous vision sensor that responds to relative intensity change2006