Serving distributed inference deep learning models in serverless computing
Authors:
- 1 July 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 109-111
Abstract
Serverless computing (SC) in an attractive win-win paradigm for cloud providers and customers, simultaneously providing greater flexibility and control over resource utilization for cloud providers while reducing costs through pay-per-use model and no capacity management for customers. While SC has been shown effective for event-triggered web applications, the use of deep learning (DL) applications on SC is limited due to latency-sensitive DL applications and stateless SC. In this paper, we focus on two key problems impacting deployment of distributed inference (DI) models on SC: resource allocation and cold start latency. To address the two problems, we propose a hybrid scheduler for identifying the optimal server resource allocation policy. The hybrid scheduler identifies container allocation based on candidate allocations from greedy strategy as well as deep reinforcement learning based allocation model.Keywords
This publication has 15 references indexed in Scilit:
- Enabling Serverless Deployment of Large-Scale AI WorkloadsIEEE Access, 2020
- Exploiting content similarity to address cold start in container deployments2019
- Optimal Pricing for Serverless Computing2019
- Memory- and Communication-Aware Model Compression for Distributed Deep Learning Inference on IoTACM Transactions on Embedded Computing Systems, 2019
- Distributed Machine Learning with a Serverless Architecture2019
- Exploring Serverless Computing for Neural Network Training2018
- Serving Deep Learning Models in a Serverless Platform2018
- Pay-Per-Request Deployment of Neural Network Models Using Serverless Architectures2018
- Towards Distributed Machine Learning in Shared Clusters: A Dynamically-Partitioned Approach2017
- Multi-task deep neural network for multi-label learning2013