Data augmentation instead of explicit regularization
Preprint
Abstract
Modern deep artificial neural networks have achieved impressive results through models with very large capacity---compared to the number of training examples---that control overfitting with the help of different forms of regularization. Regularization can be implicit, as is the case of stochastic gradient descent and parameter sharing in convolutional layers, or explicit. Most common explicit regularization techniques, such as weight decay and dropout, reduce the effective capacity of the model and typically require the use of deeper and wider architectures to compensate for the reduced capacity. Although these techniques have been proven successful in terms of improved generalization, they seem to waste capacity. In contrast, data augmentation techniques do not reduce the effective capacity and improve generalization by increasing the number of training examples. In this paper we systematically analyze the effect of data augmentation on some popular architectures and conclude that data augmentation alone---without any other explicit regularization techniques---can achieve the same performance or higher as regularized models, especially when training with fewer examples, and exhibits much higher adaptability to changes in the architecture.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Smart Augmentation Learning an Optimal Data Augmentation StrategyIEEE Access, 2017
- Improving music source separation based on deep neural networks through data augmentation and network blending2017
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Deep, Big, Simple Neural Nets for Handwritten Digit RecognitionNeural Computation, 2010
- Enhancing Text Categorization with Semantic-enriched Representation and Training Data AugmentationJournal of the American Medical Informatics Association, 2006
- Regularization Theory and Neural Networks ArchitecturesNeural Computation, 1995
- On the Uniform Convergence of Relative Frequencies of Events to Their ProbabilitiesTheory of Probability & Its Applications, 1971