Dynamics of Learning Near Singularities in Layered Networks
Authors:
- 1 March 2008
- journal article
- Published by MIT Press in Neural Computation
- Vol. 20 (3) , 813-843
Abstract
We explicitly analyze the trajectories of learning near singularities in hierarchical networks, such as multilayer perceptrons and radial basis function networks, which include permutation symmetry of hidden nodes, and show their general properties. Such symmetry induces singularities in their parameter space, where the Fisher information matrix degenerates and odd learning behaviors, especially the existence of plateaus in gradient descent learning, arise due to the geometric structure of singularity. We plot dynamic vector fields to demonstrate the universal trajectories of learning near singularities. The singularity induces two types of plateaus, the on-singularity plateau and the near-singularity plateau, depending on the stability of the singularity and the initial parameters of learning. The results presented in this letter are universally applicable to a wide class of hierarchical models. Detailed stability analysis of the dynamics of learning in radial basis function networks and multilayer perceptrons will be presented in separate work.Keywords
This publication has 32 references indexed in Scilit:
- Singularities Affect Dynamics of Learning in NeuromanifoldsNeural Computation, 2006
- Difficulty of Singularity in Population CodingNeural Computation, 2005
- Natural Gradient Works Efficiently in LearningNeural Computation, 1998
- Online Learning in Radial Basis Function NetworksNeural Computation, 1997
- Testing in locally conic models, and application to mixture modelsESAIM: Probability and Statistics, 1997
- Transient dynamics of on-line learning in two-layered neural networksJournal of Physics A: General Physics, 1996
- Learning by on-line gradient descentJournal of Physics A: General Physics, 1995
- On the Geometry of Feedforward Neural Network Error SurfacesNeural Computation, 1993
- Neural theory of association and concept-formationBiological Cybernetics, 1977
- A Theory of Adaptive Pattern ClassifiersIEEE Transactions on Electronic Computers, 1967