TRe-Map: Towards Reducing the Overheads of Fault-Aware Retraining of Deep Neural Networks by Merging Fault Maps
Authors:
- 1 September 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 434-441
Abstract
Recently, fault-aware retraining has emerged as a promising approach to improve the error resilience of Deep Neu-ral Networks (DNNs) against manufacturing-induced defects in DNN accelerators. However, state-of-the-art fault-aware training techniques incur a gigantic retraining overhead due to their per-chip retraining nature for the chip’s unique fault map, which may render it practically infeasible if retraining is done on large datasets. To address this major limitation and improve the practicability of the fault-aware retraining methodology, this work proposes a novel concept of merging fault maps to effectively retrain a DNN for a group of faulty chips in a single fault-aware retraining round. The merging of fault maps enables to avoid per chip retraining and thereby reduces the retraining overhead significantly. However, the merging of fault maps brings in new challenges such as training divergence (accuracy collapse) if a high number of accumulated faults are injected into the network in the first epoch. To address these challenges, we propose a methodology for effective merging of fault maps and then retraining of DNNs. Experimental results show that our methodology offers at least 1.4x retraining speedup on average while improving the error resilience of the network (depending on the DNN models and the number of merged fault maps). For example, for the Resnet-32 model using fault map generated from 5 fault maps at the fault rate 6e-3, our methodology offers 2x retraining speedup and 0.6% classification accuracy drop against per-chip retraining.Keywords
This publication has 16 references indexed in Scilit:
- Analyzing and mitigating the impact of permanent faults on a systolic array based neural network accelerator2018
- EIE: Efficient Inference Engine on Compressed Deep Neural Network2016
- EyerissACM SIGARCH Computer Architecture News, 2016
- Deep Residual Learning for Image Recognition2016
- Deep learningNature, 2015
- Fixed point optimization of deep convolutional neural networks for object recognition2015
- SRAM Redundancy - Silicon Area versus Number of Repairs Trade-off2008
- Trends and challenges in VLSI circuit reliabilityIEEE Micro, 2003
- On-line integrity monitoring of microprocessor control logicMicroelectronics Journal, 2001
- Learning representations by back-propagating errorsNature, 1986