A ReLU Dense Layer to Improve the Performance of Neural Networks
Authors:
- 6 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2810-2814
Abstract
We propose ReDense as a simple and low complexity way to improve the performance of trained neural networks. We use a combination of random weights and rectified linear unit (ReLU) activation function to add a ReLU dense (ReDense) layer to the trained neural network such that it can achieve a lower training loss. The lossless flow property (LFP) of ReLU is the key to achieve the lower training loss while keeping the generalization error small. ReDense does not suffer from vanishing gradient problem in the training due to having a shallow structure. We experimentally show that ReDense can improve the training and testing performance of various neural network architectures with different optimization loss and activation functions. Finally, we test ReDense on some of the state-of-the-art architectures and show the performance improvement on benchmark datasets.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- A Low Complexity Decentralized Neural Net with Centralized Equivalence using Layer-wise Learning2020
- High-Dimensional Neural Feature Using Rectified Linear Unit And Random Matrix Instance2020
- A survey on Image Data Augmentation for Deep LearningJournal of Big Data, 2019
- A Study and Comparison of Human and Deep Learning Recognition Performance under Visual Distortions2017
- Deep Residual Learning for Image Recognition2016
- Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?IEEE Transactions on Signal Processing, 2016
- On Predicting the Optimal Number of Hidden Nodes2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Deep learning in neural networks: An overviewNeural Networks, 2015
- Early Stopping — But When?2012