Are LSTMs good few-shot learners?
Open Access
- 7 September 2023
- journal article
- research article
- Published by Springer Nature in Machine Learning
- Vol. 112 (11) , 4635-4662
Abstract
Deep learning requires large amounts of data to learn new tasks well, limiting its applicability to domains where such data is available. Meta-learning overcomes this limitation by learning how to learn. Hochreiter et al. (International conference on artificial neural networks, Springer, 2001) showed that an LSTM trained with backpropagation across different tasks is capable of meta-learning. Despite promising results of this approach on small problems, and more recently, also on reinforcement learning problems, the approach has received little attention in the supervised few-shot learning setting. We revisit this approach and test it on modern few-shot learning benchmarks. We find that LSTM, surprisingly, outperform the popular meta-learning technique MAML on a simple few-shot sine wave regression benchmark, but that LSTM, expectedly, fall short on more complex few-shot image classification benchmarks. We identify two potential causes and propose a new method called Outer Product LSTM (OP-LSTM) that resolves these issues and displays substantial performance gains over the plain LSTM. Compared to popular meta-learning baselines, OP-LSTM yields competitive performance on within-domain few-shot image classification, and performs better in cross-domain settings by 0.5–1.9% in accuracy score. While these results alone do not set a new state-of-the-art, the advances of OP-LSTM are orthogonal to other advances in the field of meta-learning, yield new insights in how LSTM work in image classification, allowing for a whole range of new research directions. For reproducibility purposes, we publish all our research code publicly.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- Human-level concept learning through probabilistic program inductionScience, 2015
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification2015
- Deep learningNature, 2015
- Human-level control through deep reinforcement learningNature, 2015
- Meta-neural networks that learn by learning2003
- Meta-learning with backpropagation2002
- Learning to Learn Using Gradient Descent2001
- Lifelong Learning Algorithms1998
- Long Short-Term MemoryNeural Computation, 1997