Non‐linear activation function approximation using a REMEZ algorithm
Open Access
- 26 March 2021
- journal article
- research article
- Published by Institution of Engineering and Technology (IET) in IET Circuits, Devices & Systems
- Vol. 15 (7) , 630-640
Abstract
Here a more accurate piecewise approximation (PWA) scheme for non‐linear activation function is proposed. It utilizes a precision‐controlled recursive algorithm to predict a sub‐range; after that, the REMEZ algorithm is used to find the corresponding approximation function. The PWA realized in three ways: using first‐order functions, that is, piecewise linear model, second‐order functions (piecewise non‐linear model), and hybrid‐order model (a mixture of first‐order and second‐order functions). The hybrid‐order approximation employs the second‐order derivative of non‐linear activation function to decide the linear and non‐linear sub‐regions, correspondingly the first‐order and second‐order functions are predicted, respectively. The accuracy is compared to the present state‐of‐the‐art approximation schemes. The multi‐layer perceptron model is designed to implement XOR‐gate, and it uses an approximate activation function. The hardware utilization is measured using the TSMC 0.18‐μm library with the Synopsys Design Compiler. Result reveals that the proposed approximation scheme efficiently approximates the non‐linear activation functions.Keywords
This publication has 17 references indexed in Scilit:
- Low-error digital hardware implementation of artificial neuron activation functions and their derivativeMicroprocessors and Microsystems, 2011
- Artificial neural networks in hardware: A survey of two decades of progressNeurocomputing, 2010
- Low-error approximation of artificial neuron sigmoid function and its derivativeElectronics Letters, 2009
- Digital Hardware Implementation of a Neural System Used for Nonlinear Adaptive PredictionJournal of Computer Science, 2006
- Efficient digital implementation of the sigmoid function for reprogrammable logicIEE Proceedings - Computers and Digital Techniques, 2003
- Digital design of sigmoid approximator for artificial neural networksElectronics Letters, 2002
- Elementary function generators for neural-network emulatorsIEEE Transactions on Neural Networks, 2000
- Piecewise linear approximation applied to nonlinear function of a neural networkIEE Proceedings - Circuits, Devices and Systems, 1997
- Sigmoid generators for neural computing using piecewise approximationsIEEE Transactions on Computers, 1996
- Efficient implementation of piecewise linear activation function for digital VLSI neural networksElectronics Letters, 1989