Meta-learning with backpropagation

Authors:

Abstract
This paper introduces gradient descent methods applied to meta-leaming (leaming how to leam) in Neural Networks. Meta-leaning has been of interest in the machine leaming field for decades because of its appealing applications to intelli- gent agents, non-stationary time series, autono- mous robots, and improved leaming algorithms. Many previous neural network-based ap- proaches toward meta-leaming have been based on evolutionary methods. We show how to use gradient descent for meta-leaming in recurrent neural networks. Based on previous work on Fixed- Weight Leaming Neural Networks, we hypothesize that any recurrent network topology and its corresponding leaming algorithm(s) is a potential meta-leaming system. We tested sev- eral recurrent neural network topologies and their corresponding forms of Backpropagation for their ability to meta-leam. One of our sys- tems, based on the Long Short-Term Memory neural network developed a leaming algorithm that could leam any two-dimensional quadratic function (from a set of such functions} after only 30 training examples.