Language Modeling on a SpiNNaker2 Neuromorphic Chip
Authors:
- 22 April 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 492-496
Abstract
As large language models continue to scale in size rapidly, so too does the computational power required to run them. Event-based networks on neuromorphic devices offer a potential way to reduce energy consumption for inference significantly. However, to date, most event-based networks that can run on neuromorphic hardware, including spiking neural networks (SNNs), have not achieved task performance even on par with LSTM models for language modeling. As a result, language modeling on neuromorphic devices has seemed a distant prospect. In this work, we demonstrate the first-ever implementation of a language model on a neuromorphic device – specifically the SpiNNaker2 chip – based on a recently published event-based architecture called the EGRU. SpiNNaker2 is a many-core neuromorphic chip designed for large-scale asynchronous processing, and the EGRU is architected to leverage such hardware efficiently while maintaining competitive task performance. This implementation marks the first time a neuromorphic language model matches LSTMs, setting the stage for taking task performance to the level of large language models. We also demonstrate results on a gesture recognition task based on inputs from a DVS camera. Overall, our results showcase the feasibility of this neuro-inspired neural network in hardware, highlighting significant gains versus conventional hardware in energy efficiency for the common use case of single batch inference.Keywords
All Related Versions
Funding Information
- Ministry of Education
This publication has 16 references indexed in Scilit:
- Cerebras Architecture Deep Dive: First Look Inside the Hardware/Software Co-Design for Deep LearningIEEE Micro, 2023
- A 95.6-TOPS/W Deep Learning Inference Accelerator With Per-Vector Scaled 4-bit Quantization in 5 nmIEEE Journal of Solid-State Circuits, 2023
- Spike Transformer: Monocular Depth Estimation for Spiking Camera2022
- Spartus: A 9.4 TOp/s FPGA-Based LSTM Accelerator Exploiting Spatio-Temporal SparsityIEEE Transactions on Neural Networks and Learning Systems, 2022
- A Long Short-Term Memory for AI Applications in Spike-based Neuromorphic HardwareNature Machine Intelligence, 2022
- FPGA Acceleration of LSTM Based on Data for Test Flight2018
- Chipmunk: A systolically scalable 0.9 mm2, 3.08Gop/s/mW @ 1.2 mW accelerator for near-sensor recurrent neural network inference2018
- A Low Power, Fully Event-Based Gesture Recognition System2017
- 1.1 Computing's energy problem (and what we can do about it)2014
- A 128 X 128 120db 30mw asynchronous vision sensor that responds to relative intensity change2006