I/O Performance Evaluation of Large-Scale Deep Learning on an HPC System
Authors:
- 1 July 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 436-439
Abstract
Recently, deep learning has become important in diverse fields. Because the process requires a huge amount of computing resources, many researchers have proposed methods to utilize large-scale clusters to reduce the training time. Despite many proposals concerning the training process for large-scale clusters, there remain areas to be developed. In this study, we benchmark the performance of Intel-Caffe, which is a generalpurpose distributed deep learning framework on the Nurion supercomputer of the Korea Institute of Science and Technology Information. We particularly focus on identifying the file I/O factors that affect the performance of Intel-Caffe, as well as a performance evaluation in a container-based environment. Finally, to the best of our knowledge, we present the first benchmark results for distributed deep learning in the container-based environment for a large-scale cluster.Keywords
This publication has 8 references indexed in Scilit:
- Portable and Reusable Deep Learning Infrastructure with Containers to Accelerate Cancer Studies2018
- ImageNet Training in Minutes2018
- Plug and play bench: Simplifying big data benchmarking using containers2017
- An In-depth Performance Characterization of CPU- and GPU-based DNN Training on Modern Architectures2017
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- Singularity: Scientific containers for mobility of computePLOS ONE, 2017
- Performance analysis of container-based hadoop cluster: OpenVZ and LXC2016
- Deep Residual Learning for Image Recognition2016