MixUp as Locally Linear Out-of-Manifold Regularization
Open Access
- 17 July 2019
- journal article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 33 (01) , 3714-3722
Abstract
MixUp (Zhang et al. 2017) is a recently proposed dataaugmentation scheme, which linearly interpolates a random pair of training examples and correspondingly the one-hot representations of their labels. Training deep neural networks with such additional data is shown capable of significantly improving the predictive accuracy of the current art. The power of MixUp, however, is primarily established empirically and its working and effectiveness have not been explained in any depth. In this paper, we develop an understanding for MixUp as a form of “out-of-manifold regularization”, which imposes certain “local linearity” constraints on the model’s input space beyond the data manifold. This analysis enables us to identify a limitation of MixUp, which we call “manifold intrusion”. In a nutshell, manifold intrusion in MixUp is a form of under-fitting resulting from conflicts between the synthetic labels of the mixed-up examples and the labels of original training data. Such a phenomenon usually happens when the parameters controlling the generation of mixing policies are not sufficiently fine-tuned on the training data. To address this issue, we propose a novel adaptive version of MixUp, where the mixing policies are automatically learned from the data using an additional network and objective function designed to avoid manifold intrusion. The proposed regularizer, AdaMixUp, is empirically evaluated on several benchmark datasets. Extensive experiments demonstrate that AdaMixUp improves upon MixUp when applied to the current art of deep classification models.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Random Erasing Data AugmentationProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Shakedrop Regularization for Deep Residual LearningIEEE Access, 2019
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- Smart Augmentation Learning an Optimal Data Augmentation StrategyIEEE Access, 2017
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- Speech recognition with deep recurrent neural networks2013
- Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research GroupsIEEE Signal Processing Magazine, 2012
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998