SERF: Efficient Scheduling for Fast Deep Neural Network Serving via Judicious Parallelism
Authors:
- Feng YanFYFeng Yan
- Olatunji RuwaseOROlatunji Ruwase
- Yuxiong HeYHYuxiong He
- Evgenia SmirniESEvgenia Smirni
- 1 November 2016
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 300-311
Abstract
Deep neural networks (DNNs) has enabled a variety of artificial intelligence applications. These applications are backed by large DNN models running in serving mode on a cloud computing infrastructure. Given the compute-intensive nature of large DNN models, a key challenge for DNN serving systems is to minimize the request response latencies. This paper characterizes the behavior of different parallelism techniques for supporting scalable and responsive serving systems for large DNNs. We identify and model two important properties of DNN workloads: homogeneous request service demand, and interference among requests running concurrently due to cache/memory contention. These properties motivate the design of SERF, a dynamic scheduling framework that is powered by an interference-aware queueing-based analytical model. We evaluate SERF in the context of an image classification service using several well known benchmarks. The results demonstrate its accurate latency prediction and its ability to adapt to changing load conditions.Keywords
This publication has 28 references indexed in Scilit:
- DaDianNao: A Machine-Learning Supercomputer2014
- Optimizing cost and performance trade-offs for MapReduce job processing in the cloud2014
- Dynamic cloud service topology adaption for minimizing resources while meeting performance goalsIBM Journal of Research and Development, 2014
- Adaptive parallelism for web search2013
- Parallelism orchestration using DoPE2011
- New and old results for the M/D/c queueAEU - International Journal of Electronics and Communications, 2006
- A nonstationary poisson view of internet traffic2004
- IP Packet Delay Variation Metric for IP Performance Metrics (IPPM)2002
- On the nonstationarity of Internet trafficACM SIGMETRICS Performance Evaluation Review, 2001
- Internet Web servers: workload characterization and performance implicationsIEEE/ACM Transactions on Networking, 1997