Train Your Classifier First: Cascade Neural Networks Training from Upper Layers to Lower Layers
Open Access
- 6 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2750-2754
Abstract
Although the lower layers of a deep neural network learn features which are transferable across datasets, these layers are not transferable within the same dataset. That is, in general, freezing the trained feature extractor (the lower layers) and retraining the classifier (the upper layers) on the same dataset leads to worse performance. In this paper, for the first time, we show that the frozen classifier is transferable within the same dataset. We develop a novel top-down training method which can be viewed as an algorithm for searching for high-quality classifiers. We tested this method on automatic speech recognition (ASR) tasks and language modelling tasks. The proposed method consistently improves recurrent neural network ASR models on Wall Street Journal, self-attention ASR models on Switchboard, and AWD-LSTM language models on WikiText-2.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- Deep Cascade LearningIEEE Transactions on Neural Networks and Learning Systems, 2018
- Joint CTC-attention based end-to-end speech recognition using multi-task learning2017
- Using the Output Embedding to Improve Language Models2017
- Multilingual training of deep neural networks2013
- Unsupervised cross-lingual knowledge transfer in DNN-based LVCSR2012
- Connectionist temporal classification2006
- On the momentum term in gradient descent learning algorithmsNeural Networks, 1999
- Long Short-Term MemoryNeural Computation, 1997
- The design for the wall street journal-based CSR corpus1992
- SWITCHBOARD: telephone speech corpus for research and development1992