SignReLU neural network and its approximation ability
Preprint
Abstract
Deep neural networks (DNNs) have garnered significant attention in various fields of science and technology in recent years. Activation functions define how neurons in DNNs process incoming signals for them. They are essential for learning non-linear transformations and for performing diverse computations among successive neuron layers. In the last few years, researchers have investigated the approximation ability of DNNs to explain their power and success. In this paper, we explore the approximation ability of DNNs using a different activation function, called SignReLU. Our theoretical results demonstrate that SignReLU networks outperform rational and ReLU networks in terms of approximation performance. Numerical experiments are conducted comparing SignReLU with the existing activations such as ReLU, Leaky ReLU, and ELU, which illustrate the competitive practical performance of SignReLU.Keywords
All Related Versions
This publication has 37 references indexed in Scilit:
- Rates of approximation by ReLU shallow neural networksJournal of Complexity, 2023
- Approximation of functions from Korobov spaces by deep convolutional neural networksAdvances in Computational Mathematics, 2022
- Activation functions in deep learning: A comprehensive survey and benchmarkNeurocomputing, 2022
- Logish: A new nonlinear nonmonotonic activation function for convolutional neural networkNeurocomputing, 2021
- Parametric Deformable Exponential Linear Units for deep neural networksNeural Networks, 2020
- Optimal approximation of piecewise smooth functions using deep ReLU neural networksNeural Networks, 2018
- Hyperbolic Cross Approximation2018
- Error bounds for approximations with deep ReLU networksNeural Networks, 2017
- Learning Theory Estimates via Integral Operators and Their ApproximationsConstructive Approximation, 2007
- Approximation properties of a multilayered feedforward artificial neural networkAdvances in Computational Mathematics, 1993