In-place Activated BatchNorm for Memory-Optimized Training of DNNs
Authors:
- Samuel Rota BuloSBSamuel Rota Bulo
- Lorenzo PorziLPLorenzo Porzi
- Peter KontschiederPKPeter Kontschieder
- 1 June 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 5639-5647
Abstract
In this work we present In-Place Activated Batch Normalization (INPLACE-ABN) - a novel approach to drastically reduce the training memory footprint of modern deep neural networks in a computationally efficient way. Our solution substitutes the conventionally used succession of BatchNorm + Activation layers with a single plugin layer, hence avoiding invasive framework surgery while providing straightforward applicability for existing deep learning frameworks. We obtain memory savings of up to 50% by dropping intermediate results and by recovering required information during the backward pass through the inversion of stored forward results, with only minor increase (0.8-2%) in computation time. Also, we demonstrate how frequently used checkpointing approaches can be made computationally as efficient as INPLACE-ABN. In our experiments on image classification, we demonstrate on-par results on ImageNet-1k with state-of-the-art approaches. On the memory-demanding task of semantic segmentation, we report competitive results for COCO-Stuff and set new state-of-the-art results for Cityscapes and Mapillary Vistas. Code can be found at https://github.com/mapillary/inplace_abn.Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes2017
- Densely Connected Convolutional Networks2017
- Loss Max-Pooling for Semantic Image Segmentation2017
- The Cityscapes Dataset for Semantic Urban Scene Understanding2016
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015