Lazy Batching: An SLA-aware Batching System for Cloud Machine Learning Inference
Authors:
- 1 February 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 493-506
Abstract
In cloud ML inference systems, batching is an essential technique to increase throughput which helps optimize total-cost-of-ownership. Prior graph batching combines the individual DNN graphs into a single one, allowing multiple inputs to be concurrently executed in parallel. We observe that the coarse-grained graph batching becomes suboptimal in effectively handling the dynamic inference request traffic, leaving significant performance left on the table. This paper proposes LazyBatching, an SLA-aware batching system that considers both scheduling and batching in the granularity of individual graph nodes, rather than the entire graph for flexible batching. We show that LazyBatching can intelligently determine the set of nodes that can be efficiently batched together, achieving an average 15×, 1.5×, and 5.5 × improvement than graph batching in terms of average response time, throughput, and SLA satisfaction, respectively.Keywords
All Related Versions
Funding Information
- National Research Foundation
- KAIST
This publication has 39 references indexed in Scilit:
- From high-level deep neural models to FPGAs2016
- CnvlutinACM SIGARCH Computer Architecture News, 2016
- MinervaACM SIGARCH Computer Architecture News, 2016
- CambriconACM SIGARCH Computer Architecture News, 2016
- NeurocubeACM SIGARCH Computer Architecture News, 2016
- EIEACM SIGARCH Computer Architecture News, 2016
- TABLA: A unified template-based framework for accelerating statistical machine learning2016
- 14.5 Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks2016
- DaDianNao: A Machine-Learning Supercomputer2014
- Long Short-Term MemoryNeural Computation, 1997