Local Sigmoid Method: Non-Iterative Deterministic Learning Algorithm for Automatic Model Construction of Neural Network
Open Access
- 23 January 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 8, 20342-20362
Abstract
A non-iterative learning algorithm for artificial neural networks is an alternative to optimize the neural network parameters with extremely fast convergence time. Extreme learning machine (ELM) is one of the fastest learning algorithms based on a non-iterative method for a single hidden layer feedforward neural network (SLFN) model. ELM uses a randomization technique that requires a large number of hidden nodes to achieve the high accuracy. This leads to a large and complex model, which is slow at the inference time. Previously, we reported analytical incremental learning (AIL) algorithm, which is a compact model and a non-iterative deterministic learning algorithm, to be used as an alternative. However, AIL cannot grow its set of hidden nodes, due to the node saturation problem. Here, we describe a local sigmoid method (LSM) that is also a sufficiently compact model and a non-iterative deterministic learning algorithm to overcome both the ELM randomization and AIL node saturation problems. The LSM algorithm is based on “divide and conquer” method that divides the dataset into several subsets which are easier to optimize separately. Each subset can be associated with a local segment represented as a hidden node that preserves local information of the subset. This technique helps us to understand the function of each hidden node of the network built. Moreover, we can use such a technique to explain the function of hidden nodes learned by backpropagation, the iterative algorithm. Based on our experimental results, LSM is more accurate than other non-iterative learning algorithms and one of the most compact models.Keywords
Funding Information
- King Mongkut's Institute of Technology Ladkrabang
- Japan International Cooperation Agency
This publication has 39 references indexed in Scilit:
- Parsimonious Extreme Learning Machine Using Recursive Orthogonal Least SquaresIEEE Transactions on Neural Networks and Learning Systems, 2014
- PCA-ELM: A Robust and Pruned Extreme Learning Machine Approach Based on Principal Component AnalysisNeural Processing Letters, 2012
- Bidirectional Extreme Learning Machine for Regression Problem and Its Learning EffectivenessIEEE Transactions on Neural Networks and Learning Systems, 2012
- Approximation by neural networks with weights varying on a finite set of directionsJournal of Mathematical Analysis and Applications, 2012
- OP-ELM: Optimally Pruned Extreme Learning MachineIEEE Transactions on Neural Networks, 2009
- Generalized matrix inversion is not harder than matrix multiplicationJournal of Computational and Applied Mathematics, 2009
- A fast pruned-extreme learning machine for classification problemNeurocomputing, 2008
- Enhanced random search based incremental extreme learning machineNeurocomputing, 2008
- Universal Approximation using Incremental Constructive Feedforward Networks with Random Hidden NodesIEEE Transactions on Neural Networks, 2006
- Multilayer feedforward networks with a nonpolynomial activation function can approximate any functionNeural Networks, 1993