Soft Error Reliability Analysis of Vision Transformers
- 5 October 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Very Large Scale Integration (VLSI) Systems
- Vol. 31 (12) , 2126-2136
Abstract
Vision transformers (ViTs) that leverage self-attention mechanism have shown superior performance on many classical vision tasks compared to convolutional neural networks (CNNs) and gain increasing popularity recently. Existing ViTs’ works mainly optimize performance and accuracy, but ViTs’ reliability issues induced by soft errors in large-scale VLSI designs have generally been overlooked. In this work, we mainly study the reliability of ViTs and investigate the vulnerability from different architecture granularities ranging from models, layers, modules, and patches for the first time. The investigation reveals that ViTs with the self-attention mechanism are generally more resilient on linear computing including general matrix–matrix multiplication (GEMM) and full connection (FC) and show a relatively even vulnerability distribution across the patches. ViTs involve more fragile non-linear computing such as softmax and GELU compared to typical CNNs. With the above observations, we propose a lightweight block-wise algorithm-based fault-tolerance (LB-ABFT) approach to protect the linear computing implemented with distinct sizes of GEMM and apply a range-based protection scheme to mitigate soft errors in non-linear computing. According to our experiments, the proposed fault-tolerant approaches enhance ViTs’ accuracy significantly with minor computing overhead in the presence of various soft errors.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62174162)
- Space Trusted Computing and Electronic Information Technology Laboratory of BICE (OBCandETL-, 2022-07)
This publication has 50 references indexed in Scilit:
- Training Modern Deep Neural Networks for Memory-Fault Robustness2019
- Selective Hardening for Neural Networks in FPGAsIEEE Transactions on Nuclear Science, 2018
- Fault Tolerance for RRAM-Based Matrix Operations2018
- Ares: A framework for quantifying the resilience of deep neural networks2018
- Accurate neuron resilience prediction for a flexible reliability management in neural network accelerators2018
- Deep Residual Learning for Image Recognition2016
- ImageNet: A large-scale hierarchical image database2009
- Radiation-induced soft errors in advanced semiconductor technologiesIEEE Transactions on Device and Materials Reliability, 2005
- Designing Reliable Systems from Unreliable Components: The Challenges of Transistor Variability and DegradationIEEE Micro, 2005
- Algorithm-Based Fault Tolerance for Matrix OperationsIEEE Transactions on Computers, 1984