Meta-learning with backpropagation
Authors:
- A.S. YoungerAYA.S. Younger
- S. HochreiterSHS. Hochreiter
- P.R. ConwellPCP.R. Conwell
- 13 November 2002
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 3, 2001-2006
Abstract
This paper introduces gradient descent methods applied to meta-leaming (leaming how to leam) in Neural Networks. Meta-leaning has been of interest in the machine leaming field for decades because of its appealing applications to intelli- gent agents, non-stationary time series, autono- mous robots, and improved leaming algorithms. Many previous neural network-based ap- proaches toward meta-leaming have been based on evolutionary methods. We show how to use gradient descent for meta-leaming in recurrent neural networks. Based on previous work on Fixed- Weight Leaming Neural Networks, we hypothesize that any recurrent network topology and its corresponding leaming algorithm(s) is a potential meta-leaming system. We tested sev- eral recurrent neural network topologies and their corresponding forms of Backpropagation for their ability to meta-leam. One of our sys- tems, based on the Long Short-Term Memory neural network developed a leaming algorithm that could leam any two-dimensional quadratic function (from a set of such functions} after only 30 training examples.Keywords
This publication has 8 references indexed in Scilit:
- A neural network that embeds its own meta-levels2002
- Learning algorithms and fixed dynamics2002
- Evolution and design of distributed learning rules2002
- Fixed-weight on-line learningIEEE Transactions on Neural Networks, 1999
- Long Short-Term MemoryNeural Computation, 1997
- The Evolution of Learning: An Experiment in Genetic Connectionism1991
- Fixed-weight networks can learn1990
- A Learning Algorithm for Continually Running Fully Recurrent Neural NetworksNeural Computation, 1989