Error Resilient Transformers: A Novel Soft Error Vulnerability Guided Approach to Error Checking and Suppression
Authors:
- 22 May 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-6
Abstract
Transformer networks have achieved remarkable success in Natural Language Processing (NLP) and Computer Vision applications. However, the underlying large volumes of Transformer computations demand high reliability and resilience to soft errors in processor hardware. The objective of this research is to develop efficient techniques for design of error resilient Transformer architectures. To enable this, we first perform a soft error vulnerability analysis of every fully connected layers in Transformer computations. Based on this study, error detection and suppression modules are selectively introduced into datapaths to restore Transformer performance under anticipated error rate conditions. Memory access errors and neuron output errors are detected using checksums of linear Transformer computations. Correction consists of determining output neurons with out-of-range values and suppressing the same to zero. For a Transformer with nominal BLEU score of 52.7, such vulnerability guided selective error suppression can recover language translation performance from a BLEU score of 0 to 50.774 with as much as 0.001 probability of activation error, incurring negligible memory and computation overheads.Keywords
Funding Information
- National Science Foundation
This publication has 9 references indexed in Scilit:
- Soft Error Resilient Deep Learning Systems Using Neuron Gradient Statistics2022
- A Low-cost Fault Corrector for Deep Neural Networks through Range Restriction2021
- Just say zero2020
- PyTorchFI: A Runtime Perturbation Tool for DNNs2020
- Sanity-Check: Boosting the Reliability of Safety-Critical Deep Neural Network Applications2019
- Error Resilient Neuromorphic Networks Using Checker Neurons2018
- Analyzing and mitigating the impact of permanent faults on a systolic array based neural network accelerator2018
- Marian: Fast Neural Machine Translation in C++2018
- Fault-Tolerant Training with On-Line Fault Detection for RRAM-Based Neural Computing Systems2017