ECML: Improving Efficiency of Machine Learning in Edge Clouds
Authors:
- 9 November 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-6
Abstract
Edge cloud data centers (Edge) are deployed to provide responsive services to the end-users. Edge can host more powerful CPUs and DNN accelerators such as GPUs and may be used for offloading tasks from end-user devices that require more significant compute capabilities. But Edge resources may also be limited and must be shared across multiple applications that process requests concurrently from several clients. However, multiplexing GPUs across applications is challenging. With edge cloud servers needing to process a lot of streaming and the advent of multi-GPU systems, getting that data from the network to the GPU can be a bottleneck, limiting the amount of work the GPU cluster can do. The lack of prompt notification of job completion from the GPU can also result in poor GPU utilization. We build on our recent work on controlled spatial sharing of a single GPU to expand to support multi-GPU systems and propose a framework that addresses these challenges. Unlike the state-of-the-art uncontrolled spatial sharing currently available with systems such as CUDA-MPS, our controlled spatial sharing approach uses each of the GPU in the cluster efficiently by removing interference between applications, resulting in much better, predictable, inference latency We also use each of the cluster GPU's DMA engines to offload data transfers to the GPU complex, thereby preventing the CPU from being the bottleneck. Finally, our framework uses the CUDA event library to give timely, low overhead GPU notifications. Our evaluations show we can achieve low DNN inference latency and improve DNN inference throughput by at least a factor of 2.Keywords
This publication has 8 references indexed in Scilit:
- GSLICE2020
- Edge Assisted Real-time Object Detection for Mobile Augmented Reality2019
- Deep Learning With Edge Computing: A ReviewProceedings of the IEEE, 2019
- NetML: An NFV Platform with Efficient Support for Machine Learning Applications2019
- DeepDecision: A Mobile Deep Learning Framework for Edge Video Analytics2018
- Edge computing for situational awareness2017
- CNTK2016
- Bringing the cloud to the edge2014