Jellyfish: Timely Inference Serving for Dynamic Edge Networks
Authors:
- 1 December 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 277-290
Abstract
While high accuracy is of paramount importance for deep learning (DL) inference, serving inference requests on time is equally critical but has not been carefully studied especially when the request has to be served over a dynamic wireless network at the edge. In this paper, we propose Jellyfish—a novel edge DL inference serving system that achieves soft guarantees on end-to-end inference latency often specified as a service-level objective (SLO). To handle the network variability, Jellyfish exploits both data and deep neural network (DNN) adaptation to conduct tradeoffs between accuracy and latency. Jellyfish features a new design that enables collective adaptation policies where the decisions for data and DNN adaptations are aligned and coordinated among multiple users with varying network conditions. We propose efficient algorithms to dynamically adapt DNNs and map users, so that we fulfill latency SLOs while maximizing the overall inference accuracy. Our experiments based on a prototype implementation and real-world WiFi and LTE network traces show that Jellyfish can meet latency SLOs at around the 99th percentile while maintaining high accuracy.Keywords
This publication has 30 references indexed in Scilit:
- Chameleon2018
- An empirical study of latency in an emerging class of edge computing applications for wearable cognitive assistance2017
- Real-Time Video Analytics: The Killer App for Edge ComputingComputer, 2017
- HTTP/2-Based Adaptive Streaming of HEVC Video Over 4G/LTE NetworksIEEE Communications Letters, 2016
- MCDNN2016
- Deep Residual Learning for Image Recognition2016
- Is image super-resolution helpful for other vision tasks?2016
- A Control-Theoretic Approach for Dynamic Adaptive Video Streaming over HTTPACM SIGCOMM Computer Communication Review, 2015
- Microsoft COCO: Common Objects in Context2014
- A close examination of performance and power characteristics of 4G LTE networks2012