Augment Your Batch: Improving Generalization Through Instance Repetition
Authors:
- 1 June 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8126-8135
Abstract
Large-batch SGD is important for scaling training of deep neural networks. However, without fine-tuning hyperparameter schedules, the generalization of the model may be hampered. We propose to use batch augmentation: replicating instances of samples within the same batch with different data augmentations. Batch augmentation acts as a regularizer and an accelerator, increasing both generalization and performance scaling for a fixed budget of optimization steps. We analyze the effect of batch augmentation on gradient variance and show that it empirically improves convergence for a wide variety of networks and datasets. Our results show that batch augmentation reduces the number of necessary SGD updates to achieve the same accuracy as the state-of-the-art. Overall, this simple yet effective method enables faster training and better generalization by allowing more computational resources to be used concurrently.Keywords
This publication has 8 references indexed in Scilit:
- Demystifying Parallel and Distributed Deep LearningACM Computing Surveys, 2019
- RenderGAN: Generating Realistic Labeled DataFrontiers in Robotics and AI, 2018
- Deep learning at 15PF2017
- Densely Connected Convolutional Networks2017
- Deep Residual Learning for Image Recognition2016
- Rethinking the Inception Architecture for Computer Vision2016
- ImageNet: A large-scale hierarchical image database2009
- Scheduling multithreaded computations by work stealingJournal of the ACM, 1999