Morphling
Authors:
Open Access
- 1 November 2021
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 639-653
Abstract
Machine learning models are widely deployed in production cloud to provide online inference services. Efficiently deploying inference services requires careful tuning of hardware and runtime configurations (e.g., GPU type, GPU memory, batch size), which can significantly improve the model serving performance and reduce cost. However, existing autoconfiguration approaches for general workloads, such as Bayesian optimization and white-box prediction, are inefficient in navigating the high-dimensional configuration space of model serving, incurring high sampling cost. In this paper, we present Morphling, a fast, near-optimal auto-configuration framework for cloud-native model serving. Morphling employs model-agnostic meta-learning to navigate the large configuration space. It trains a metamodel offline to capture the general performance trend under varying configurations. Morphling quickly adapts the metamodel to a new inference service by sampling a small number of configurations and uses it to find the optimal one. We have implemented Morphling as an auto-configuration service in Kubernetes, and evaluate its performance with popular CV and NLP models, as well as the production inference services in Alibaba. Compared with existing approaches, Morphling reduces the median search cost by 3x-22x, quickly converging to the optimal configuration by sampling only 30 candidates in a large search space consisting of 720 options.Keywords
This publication has 20 references indexed in Scilit:
- Swayam2017
- Selecting the best VM across multiple public clouds2017
- Google Vizier2017
- SERF: Efficient Scheduling for Fast Deep Neural Network Serving via Judicious Parallelism2016
- Learning to Learn: Model Regression Networks for Easy Small Sample Learning2016
- Deep Networks with Stochastic Depth2016
- Identity Mappings in Deep Residual Networks2016
- Deep Residual Learning for Image Recognition2016
- Efficient One-Pass Decoding with NNLM for Speech RecognitionIEEE Signal Processing Letters, 2014
- Jockey2012