A Universal Activation Function for Deep Learning
Authors:
Open Access
- 1 January 2023
- journal article
- research article
- Published by Tech Science Press in Computers, Materials & Continua
- Vol. 75 (2) , 3553-3569
Abstract
Recently, deep learning has achieved remarkable results in fields that require human cognitive ability, learning ability, and reasoning ability. Activation functions are very important because they provide the ability of artificial neural networks to learn complex patterns through nonlinearity. Various activation functions are being studied to solve problems such as vanishing gradients and dying nodes that may occur in the deep learning process. However, it takes a lot of time and effort for researchers to use the existing activation function in their research. Therefore, in this paper, we propose a universal activation function (UA) so that researchers can easily create and apply various activation functions and improve the performance of neural networks. UA can generate new types of activation functions as well as functions like traditional activation functions by properly adjusting three hyperparameters. The famous Convolutional Neural Network (CNN) and benchmark dataset were used to evaluate the experimental performance of the UA proposed in this study. We compared the performance of the artificial neural network to which the traditional activation function is applied and the artificial neural network to which the UA is applied. In addition, we evaluated the performance of the new activation function generated by adjusting the hyperparameters of the UA. The experimental performance evaluation results showed that the classification performance of CNNs improved by up to 5% through the UA, although most of them showed similar performance to the traditional activation function.Keywords
This publication has 11 references indexed in Scilit:
- ReID-DeePNet: A Hybrid Deep Learning System for Person Re-IdentificationMathematics, 2022
- Deep Learning Approach Based on Residual Neural Network and SVM Classifier for Driver’s Distraction DetectionApplied Sciences, 2022
- RMAF: Relu-Memristor-Like Activation Function for Deep LearningIEEE Access, 2020
- Improved Convolutional Neural Network Based on Fast Exponentially Linear Unit Activation FunctionIEEE Access, 2019
- Dual Rectified Linear Units (DReLUs): A replacement for tanh activation functions in Quasi-Recurrent Neural NetworksPattern Recognition Letters, 2018
- Complex-Valued Neural Networks With Nonparametric Activation FunctionsIEEE Transactions on Emerging Topics in Computational Intelligence, 2018
- A novel softplus linear unit for deep convolutional neural networksApplied Intelligence, 2017
- On the approximation of the step function by some sigmoid functionsMathematics and Computers in Simulation, 2017
- On general form of the Tanh method and its application to nonlinear partial differential equationsNumerical Algebra, Control & Optimization, 2016
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014