Benchmarking Deep Learning Frameworks: Design Considerations, Metrics and Beyond
Authors:
- Ling LiuLLLing Liu
- Yanzhao WuYWYanzhao Wu
- Wenqi WeiWWWenqi Wei
- Wenqi CaoWCWenqi Cao
- Semih SahinSSSemih Sahin
- Qi ZhangQZQi Zhang
- 1 July 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1258-1269
Abstract
With increasing number of open-source deep learning (DL) software tools made available, benchmarking DL software frameworks and systems is in high demand. This paper presents design considerations, metrics and challenges towards developing an effective benchmark for DL software frameworks and illustrate our observations through a comparative study of three popular DL frameworks: TensorFlow, Caffe, and Torch. First, we show that these deep learning frameworks are optimized with their default configurations settings. However, the default configuration optimized on one specific dataset may not work effectively for other datasets with respect to runtime performance and learning accuracy. Second, the default configuration optimized on a dataset by one DL framework does not work well for another DL framework on the same dataset. Third, experiments show that different DL frameworks exhibit different levels of robustness against adversarial examples. Through this study, we envision that unlike traditional performance-driven benchmarks, benchmarking deep learning software frameworks should take into account of both runtime and accuracy and their latent interaction with hyper-parameters and data-dependent configurations of DL frameworks.Keywords
This publication has 13 references indexed in Scilit:
- An In-depth Performance Characterization of CPU- and GPU-based DNN Training on Modern Architectures2017
- Performance Evaluation of Deep Learning Tools in Docker Containers2017
- Evaluation of Deep Learning Frameworks Over Different HPC Architectures2017
- Towards Evaluating the Robustness of Neural Networks2017
- Performance analysis of CNN frameworks for GPUs2017
- The Limitations of Deep Learning in Adversarial Settings2016
- Efficient Machine Learning for Big Data: A ReviewBig Data Research, 2015
- Deep neural networks are easily fooled: High confidence predictions for unrecognizable images2015
- Big Data Deep Learning: Challenges and PerspectivesIEEE Access, 2014
- Stochastic Gradient Descent Tricks2012