A random energy approach to deep learning
Authors:
- 1 July 2022
- journal article
- research article
- Published by IOP Publishing in Journal of Statistical Mechanics: Theory and Experiment
- Vol. 2022 (7) , 073404
Abstract
We study a generic ensemble of deep belief networks (DBN) which is parametrized by the distribution of energy levels of the hidden states of each layer. We show that, within a random energy approach, statistical dependence can propagate from the visible to deep layers only if each layer is tuned close to the critical point during learning. As a consequence, efficiently trained learning machines are characterised by a broad distribution of energy levels. The analysis of DBNs and restricted Boltzmann machines on different datasets confirms these conclusions.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Learning with hidden variablesCurrent Opinion in Neurobiology, 2015
- Statistical Mechanics of the US Supreme CourtJournal of Statistical Physics, 2015
- On sampling and modeling complex systemsJournal of Statistical Mechanics: Theory and Experiment, 2013
- The Functional Benefits of Criticality in the CortexThe Neuroscientist, 2012
- Are Biological Systems Poised at Criticality?Journal of Statistical Physics, 2011
- The criticality hypothesis: how local cortical networks might optimize information processingPhilosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2007
- Real-Time Computation at the Edge of Chaos in Recurrent Neural NetworksNeural Computation, 2004
- Computation at the edge of chaos: Phase transitions and emergent computationPhysica D: Nonlinear Phenomena, 1990
- Random-energy model: An exactly solvable model of disordered systemsPhysical Review B, 1981
- Will a Large Complex System be Stable?Nature, 1972