Benchmarking Inference of Transformer-Based Transcription Models With Clustering on Embedded GPUs
Open Access
- 11 July 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 12, 123276-123293
Abstract
Early awareness of inference performance ensures the feasibility of machine learning for embedded deployment. Often, ML model selection often focuses first on training performance and accuracy, with inference considered second. While prioritizing training is necessary, model inference performance is often impacted by these optimizations. Knowing whether a model will run meaningfully faster or be more energy efficient than its un-optimized-counterpart is required for resource-constrained embedded environments. Training-time optimizations may incur real performance losses at inference time. This paper benchmarks the effect of one such training optimization, clustered attention, and examines its effect on the inference performance of the transformer-based transcription model wav2vec2. The execution time and energy consumption of this model is evaluated on NVIDIA Jetson-embedded GPU devices. Clustered attention is specific to the transformer self-attention mechanism, which exhibits poor memory and execution-time scaling with respect to a variable input size. These scaling characteristics make it a potentially critical bottleneck that must be observed under realistic conditions. Our research considers three model variants: reference (original), clustered, and improved-clustered attention models. The reference model was faster for small input sizes, but the clustered model was faster for input sizes longer than 10 seconds. The clustered model had the lowest maximum energy per inference for input sizes greater than about 12 seconds of audio. With optimal configuration, the improved-clustered model takes 26.34% more time to execute than the reference model. With these operational differences, we show that inference performance and energy consumption of deployment should not be overlooked in selecting model optimizations.Keywords
Funding Information
- NSF Space, High-Performance, and REsilient Computing Center’s industry and agency members and by the IUCRC Program of the National Science Foundation (CNS-1738783)
This publication has 16 references indexed in Scilit:
- Demystifying BERT: System Design Implications2022
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden UnitsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- Benchmarking Transformer-Based Transcription on Embedded GPUs for Space Applications2021
- Self-Training and Pre-Training are Complementary for Speech Recognition2021
- schuBERT: Optimizing Elements of BERT2020
- wav2vec: Unsupervised Pre-Training for Speech Recognition2019
- English Conversational Telephone Speech Recognition by Humans and Machines2017
- Librispeech: An ASR corpus based on public domain audio books2015
- Natural language processing: an introductionJournal of the American Medical Informatics Association, 2011
- Connectionist temporal classification2006