Just say zero
Authors:
Open Access
- 2 November 2020
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-9
Abstract
DNNs are abundantly employed in a variety of applications, including real-time systems with strict safety constraints. The consequences of errors prove disastrous in safety-critical systems, such as autonomous driving, healthcare, and industrial applications. DNNs are resilient to limited numerical perturbations yet fragile under large deviations in weights and activations. The traditional error tolerance measures fail to meet the tight design constraints of DNN processing systems due to extensive overheads or limited advantages in abundant error conditions. The algorithmic particularities of DNNs though create novel opportunities to deal with errors more effectively and economically. We revisit the two fundamental tasks in fault-tolerant system design, namely, error detection and correction, and demonstrate that the precise versions of these operations could be replaced by approximated counterparts in DNNs to deliver an extensive bit-error resilience even at high error rates while necessitating no information redundancy. We first maintain DNN accuracy even under extreme error rates by suppressing the numerical contributions of anomalous activations, eliminating any reliance on precise error correction. We tackle the problem of no redundancy error detection by establishing in training numerical associations among activations, and employing them for anomaly detection. Anomalous feature detection and suppression, performed efficiently at inference with minimal resources in a DNN accelerator, is shown to deliver significant resilience boosts while imposing neither information redundancy nor perceptible overheads.Keywords
This publication has 24 references indexed in Scilit:
- Understanding error propagation in deep learning neural network (DNN) accelerators and applications2017
- Rescuing Memristor-based Neuromorphic Design with High Defects2017
- Accelerator-friendly neural-network training: Learning variations and defects in RRAM crossbar2017
- MinervaACM SIGARCH Computer Architecture News, 2016
- Deep Residual Learning for Image Recognition2016
- Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognitionNeural Networks, 2012
- Razor: circuit-level correction of timing errors for low-power operationIEEE Micro, 2004
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Algorithm-Based Fault Tolerance for Matrix OperationsIEEE Transactions on Computers, 1984
- Error Detecting and Error Correcting CodesBell System Technical Journal, 1950