INFless: a native serverless system for low-latency, high-throughput inference
Authors:
Top Cited Papers
Open Access
- 22 February 2022
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 768-781
Abstract
Modern websites increasingly rely on machine learning (ML) to improve their business efficiency. Developing and maintaining ML services incurs high costs for developers. Although serverless systems are a promising solution to reduce costs, we find that the current general purpose serverless systems cannot meet the low latency, high throughput demands of ML services. While simply ”patching” general serverless systems does not resolve the problem completely, we propose that such a system should natively combine the features of inference with a serverless paradigm. We present INFless, the first ML domain-specific serverless platform. It provides a unified, heterogeneous resource abstraction between CPU and accelerators, and achieves high throughput using built-in batching and non-uniform scaling mechanisms. It also supports low latency through coordinated management of batch queuing time, execution time and coldstart rate. We evaluate INFless using both a local cluster testbed and a large-scale simulation. Experimental results show that INFless outperforms state-of-the-art systems by 2×-5× on system throughput, meeting the latency goals of ML services.Keywords
This publication has 16 references indexed in Scilit:
- MLPerf: An Industry Standard Benchmark Suite for Machine Learning PerformanceIEEE Micro, 2020
- Face Recognition Based on The Improved MobileNet2019
- Nexus2019
- Text Classification Based on TextCNN for Power Grid User Fault Repairing Information2018
- Swayam2017
- Question answering over knowledgebase with attention-based LSTM networks and knowledge embeddings2017
- DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Fathom: reference workloads for modern deep learning methods2016
- Deep Residual Learning for Image Recognition2016
- Deep Face Recognition2015