Active Identity Function as Activation Function
Preprint
- 15 May 2023
- preprint
- Published by MDPI AG in Preprints.org
Abstract
Selecting the optimal activation function for training deep neural networks has always been challenging because it significantly impacts the neural network’s performance and training speed. At this point, researchers are more likely to employ RELU than other well-known activation functions. After RELU, researchers have proposed many activation functions to improve RELU. None of them was capable of surpassing RELU as their most significant rival. SWISH outperformed RELU in several challenging experiments like classification, object detection, and tracking. Replacing RELU units with SWISH, which improves performance in many tasks. This paper proposed a new activation function surpassing Google’s brain’s SWISH function, Which we named AIF. Experiments indicate that our proposed activation function outperforms SWISH in various tasks.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Reminder of the First Paper on Transfer Learning in Neural Networks, 1976Informatica, 2020
- Searching for Activation FunctionsarXiv, 2017
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision ApplicationsarXiv, 2017
- Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)arXiv, 2016
- Deep learningNature, 2015
- What is the best multi-stage architecture for object recognition?2009
- ImageNet: A large-scale hierarchical image database2009
- Digital selection and analogue amplification coexist in a cortex-inspired silicon circuitNature, 2000
- Gradient Theory of Optimal Flight PathsARS Journal, 1960
- A logical calculus of the ideas immanent in nervous activityBulletin of Mathematical Biology, 1943