ZNN -- A Fast and Scalable Algorithm for Training 3D Convolutional Networks on Multi-core and Many-Core Shared Memory Machines
Authors:
- Aleksandar ZlateskiAZAleksandar Zlateski
- Kisuk LeeKLKisuk Lee
- H. Sebastian SeungHSH. Sebastian Seung
- 1 May 2016
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 15302075,p. 801-811
Abstract
Convolutional networks (ConvNets) have become a popular approach to computer vision. It is important to accelerate ConvNet training, which is computationally costly. We propose a novel parallel algorithm based on decomposition into a set of tasks, most of which are convolutions or FFTs. Applying Brent's theorem to the task dependency graph implies that linear speedup with the number of processors is attainable within the PRAM model of parallel computation, for wide network architectures. To attain such performance on real shared-memory machines, our algorithm computes convolutions converging on the same node of the network with temporal locality to reduce cache misses, and sums the convergent convolution outputs via an almost wait-free concurrent method to reduce time spent in critical sections. We implement the algorithm with a publicly available software package called ZNN. Benchmarking with multi-core CPUs shows that ZNN can attain speedup roughly equal to the number of physical cores. We also show that ZNN can attain over 90× speedup on a many-core CPU (Xeon Phi™ Knights Corner). These speedups are achieved for network architectures with widths that are in common use. The task parallelism of the ZNN algorithm is suited to CPUs, while the SIMD parallelism of previous algorithms is compatible with GPUs. Through examples, we show that ZNN can be either faster or slower than certain GPU implementations depending on specifics of the network architecture, kernel sizes, and density and size of the output patch. ZNN may be less costly to develop and maintain, due to the relative ease of general-purpose CPU programming.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- The Potential of the Intel (R) Xeon Phi for Supervised Deep Learning2015
- Parallelizing Convolutional Neural Networks on Intel$^{\textregistered }$ Many Integrated Core Architecture2015
- Caffe2014
- Training Large Scale Deep Neural Networks on the Intel Xeon Phi Many-Core Coprocessor2014
- Fast image scanning with deep max-pooling convolutional neural networks2013
- A fast learning algorithm for image segmentation with max-pooling convolutional networks2013
- Connectomic reconstruction of the inner plexiform layer in the mouse retinaNature, 2013
- Multicore and GPU Parallelization of Neural Networks for Face RecognitionProcedia Computer Science, 2013
- Traffic sign recognition with multi-scale Convolutional Networks2011
- Scheduling multithreaded computations by work stealingJournal of the ACM, 1999