Continual and Multi-Task Architecture Search
Authors:
Preprint
Abstract
Architecture search is the process of automatically learning the neural model or cell structure that best suits the given task. Recently, this approach has shown promising performance improvements (on language modeling and image classification) with reasonable training speed, using a weight sharing strategy called Efficient Neural Architecture Search (ENAS). In our work, we first introduce a novel continual architecture search (CAS) approach, so as to continually evolve the model parameters during the sequential training of several tasks, without losing performance on previously learned tasks (via block-sparsity and orthogonality constraints), thus enabling life-long learning. Next, we explore a multi-task architecture search (MAS) approach over ENAS for finding a unified, single cell structure that performs well across multiple tasks (via joint controller rewards), and hence allows more generalizable transfer of the cell structure knowledge to an unseen new task. We empirically show the effectiveness of our sequential continual learning and parallel multi-task learning based architecture search approaches on diverse sentence-pair classification tasks (GLUE) and multimodal-generation based video captioning tasks. Further, we present several ablations and analyses on the learned cell structures.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Learning Transferable Architectures for Scalable Image Recognition2018
- Efficient Architecture Search by Network TransformationProceedings of the AAAI Conference on Artificial Intelligence, 2018
- Aggregated Residual Transformations for Deep Neural Networks2017
- Group sparse regularization for deep neural networksNeurocomputing, 2017
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences of the United States of America, 2017
- Deep Residual Learning for Image Recognition2016
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language2016
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016
- CIDEr: Consensus-based image description evaluation2015
- The Inference of Regular LISP Programs from ExamplesIEEE Transactions on Systems, Man, and Cybernetics, 1978