Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models
Open Access
- 24 March 2024
- journal article
- research article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 38 (11) , 11996-12005
Abstract
Current AI training infrastructure is dominated by single instruction multiple data (SIMD) and systolic array architectures, such as Graphics Processing Units (GPUs) and Tensor Processing Units (TPUs), that excel at accelerating parallel workloads and dense vector matrix multiplications. Potentially more efficient neural network models utilizing sparsity and recurrence cannot leverage the full power of SIMD processor and are thus at a severe disadvantage compared to today's prominent parallel architectures like Transformers and CNNs, thereby hindering the path towards more sustainable AI. To overcome this limitation, we explore sparse and recurrent model training on a massively parallel multiple instruction multiple data (MIMD) architecture with distributed local memory. We implement a training routine based on backpropagation though time (BPTT) for the brain-inspired class of Spiking Neural Networks (SNNs) that feature binary sparse activations. We observe a massive advantage in using sparse activation tensors with a MIMD processor, the Intelligence Processing Unit (IPU) compared to GPUs. On training workloads, our results demonstrate 5-10x throughput gains compared to A100 GPUs and up to 38x gains for higher levels of activation sparsity, without a significant slowdown in training convergence or reduction in final model performance. Furthermore, our results show highly promising trends for both single and multi IPU configurations as we scale up to larger model sizes. Our work paves the way towards more efficient, non-standard models via AI training hardware beyond GPUs, and competitive large scale SNN models.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- Tricking AI chips into simulating the human brain: A detailed performance analysisNeurocomputing, 2024
- ReckOn: A 28nm Sub-mm2 Task-Agnostic Spiking Recurrent Neural Network Processor Enabling On-Chip Learning over Second-Long Timescales2022
- Visualizing a joint future of neuroscience and neuromorphic engineeringNeuron, 2021
- Larger GPU-accelerated brain simulations with procedural connectivityNature Computational Science, 2021
- A solution to the learning dilemma for recurrent networks of spiking neuronsNature Communications, 2020
- Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High-Speed ScenariosIEEE Robotics and Automation Letters, 2018
- GeNN: a code generation framework for accelerated brain simulationsScientific Reports, 2016
- Neuromorphic Electronic Circuits for Building Autonomous Cognitive SystemsProceedings of the IEEE, 2014
- Neurogrid: A Mixed-Analog-Digital Multichip System for Large-Scale Neural SimulationsProceedings of the IEEE, 2014
- 1.1 Computing's energy problem (and what we can do about it)2014