High-Dimensional Neural Feature Using Rectified Linear Unit And Random Matrix Instance
Authors:
- 1 May 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4237-4241
Abstract
We design a ReLU-based multilayer neural network to generate a rich high-dimensional feature vector. The feature guarantees a monotonically decreasing training cost as the number of layers increases. We design the weight matrix in each layer to extend the feature vectors to a higher dimensional space while providing a richer representation in the sense of training cost. Linear projection to the target in the higher dimensional space leads to a lower training cost if a convex cost is minimized. An ` 2 -norm convex constraint is used in the minimization to improve the generalization error and avoid overfitting. The regularization hyperparameters of the network are derived analytically to guarantee a monotonic decrement of the training cost and therefore, it eliminates the need for cross-validation to find the regularization hyperparameter in each layer.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Experimental Study on Extreme Learning Machine Applications for Speech EnhancementIEEE Access, 2017
- A Study and Comparison of Human and Deep Learning Recognition Performance under Visual Distortions2017
- Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?IEEE Transactions on Signal Processing, 2016
- Hierarchical Extreme Learning Machine for unsupervised representation learning2015
- Extreme Learning Machine for Multilayer PerceptronIEEE Transactions on Neural Networks and Learning Systems, 2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Trends in extreme learning machines: A reviewNeural Networks, 2015
- Invariant Scattering Convolution NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Extreme Learning Machine for Regression and Multiclass ClassificationIEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 2011
- Distributed Optimization and Statistical Learning via the Alternating Direction Method of MultipliersFoundations and Trends in Machine Learning, 2011