Aggressive Approximation of the SoftMax Function for Power-Efficient Hardware Implementations
- 15 October 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems II: Express Briefs
- Vol. 69 (3) , 1652-1656
Abstract
Neural Network models most often exploit the SoftMax function in the classification stage for computing probabilities through exponentiation and division operations. To reduce the complexity and the energy consumption of such stage, several hardware-friendly approximation strategies have been disclosed in the recent past. This brief evaluates the effects of an aggressive approximation of the SoftMax layer on both classification accuracy and hardware characteristics. Experimental results demonstrate that the proposed circuit, when implemented in a 28 nm FDSOI technology, saves 65% of silicon area with respect to competitors, dissipating less than 1 pJ. FPGA implementation results confirm a massive energy dissipation reduction with respect to the conventional baseline architecture, without introducing penalties in the Top-1 accuracy.Keywords
This publication has 16 references indexed in Scilit:
- A Multi-Class Objects Detection Coprocessor With Dual Feature Space and Weighted SoftmaxIEEE Transactions on Circuits and Systems II: Express Briefs, 2020
- Efficient Precision-Adjustable Architecture for Softmax Function in Deep LearningIEEE Transactions on Circuits and Systems II: Express Briefs, 2020
- A High-Accuracy Implementation for Softmax Layer in Deep Neural Networks2020
- A Survey of Accelerator Architectures for Deep Neural NetworksEngineering, 2020
- Energy-Efficient Architecture for CNNs Inference on Heterogeneous FPGAJournal of Low Power Electronics and Applications, 2019
- Efficient Softmax Hardware Architecture for Deep Neural Networks2019
- Real-Time Speech Recognition for IoT Purpose using a Delta Recurrent Neural Network Accelerator2019
- Scaling equations for the accurate prediction of CMOS device performance from 180 nm to 7 nmIntegration, 2017
- Efficient hardware architecture of softmax layer in deep neural network2016
- Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition1990