Evolving Fully Automated Machine Learning via Life-Long Knowledge Anchors
- 29 March 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 43 (9) , 3091-3107
Abstract
Automated machine learning (AutoML) has achieved remarkable progress on various tasks, which is attributed to its minimal involvement of manual feature and model designs. However, most of existing AutoML pipelines only touch parts of the full machine learning pipeline, e.g., neural architecture search or optimizer selection. This leaves potentially important components such as data cleaning and model ensemble out of the optimization, and still results in considerable human involvement and suboptimal performance. The main challenges lie in the huge search space assembling all possibilities over all components, as well as the generalization ability over different tasks like image, text, and tabular etc . In this paper, we present a first-of-its-kind fully AutoML pipeline, to comprehensively automate data preprocessing, feature engineering, model generation/selection/training and ensemble for an arbitrary dataset and evaluation metric. Our innovation lies in the comprehensive scope of a learning pipeline, with a novel “life-long” knowledge anchor design to fundamentally accelerate the search over the full search space. Such knowledge anchors record detailed information of pipelines and integrates them with an evolutionary algorithm for joint optimization across components. Experiments demonstrate that the result pipeline achieves state-of-the-art performance on multiple datasets and modalities. Specifically, the proposed framework was extensively evaluated in the NeurIPS 2019 AutoDL challenge, and won the only champion with a significant gap against other approaches, on all the image, video, speech, text and tabular tracks.Keywords
Funding Information
- National Science Fund for Distinguished Young Scholars (62025603)
- National Natural Science Foundation of China (U1705262, 62072386, 62072387, 62072389, 62002305, 61772443, 61802324, 61702136)
- Basic and Applied Basic Research Foundation of Guangdong Province (2019B1515120049)
This publication has 44 references indexed in Scilit:
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Multi-class geospatial object detection and geographic image classification based on collection of part detectorsISPRS Journal of Photogrammetry and Remote Sensing, 2014
- Convolutional Neural Networks for Sentence Classification2014
- Auto-WEKA2013
- Meta-learning for evolutionary parameter optimization of classifiersMachine Learning, 2012
- Distinctive Image Features from Scale-Invariant KeypointsInternational Journal of Computer Vision, 2004
- A lifelong learning perspective for mobile robot control2002
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Characterization of classification algorithms1995
- An evolutionary algorithm that constructs recurrent neural networksIEEE Transactions on Neural Networks, 1994