TCB: Accelerating Transformer Inference Services with Request Concatenation
Authors:
Open Access
- 29 August 2022
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-11
Abstract
Transformer has dominated the field of natural language processing because of its strong capability in learning from sequential input data. In recent years, various computing and networking optimizations have been proposed for improving transformer training efficiency. However, transformer inference, as the core of many AI services, has been seldom studied. A key challenge of transformer inference is variable-length input. In order to align these input, existing work has proposed batching schemes by padding zeros, which unfortunately introduces significant computational redundancy. Moreover, existing transformer inference studies are separated from the whole serving system, where both request batching and request scheduling are critical and they have complex interaction. To fill the research gap, we propose TCB, a Transformer inference system with a novel ConcatBatching scheme as well as a jointly designed online scheduling algorithm. ConcatBatching minimizes computational redundancy by concatenating multiple requests, so that batch rows can be aligned with reduced padded zeros. Moreover, we conduct a systemic study by designing an online request scheduling algorithm aware of ConcatBatching. This scheduling algorithm needs no future request information and has provable theoretical guarantee. Experimental results show that TCB can significantly outperform state-of-the-art.Keywords
Funding Information
- Japan Society for the Promotion of Science KAKENHI (21H03424)
This publication has 15 references indexed in Scilit:
- Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers2021
- ELSA: Hardware-Software Co-design for Efficient, Lightweight Self-Attention Mechanism in Neural Networks2021
- Interference-Aware Scheduling for Inference Serving2021
- TurboTransformers2021
- Lazy Batching: An SLA-aware Batching System for Cloud Machine Learning Inference2021
- GSLICE2020
- Automating Cloud Deployment for Deep Learning Inference of Real-time Online Services2020
- PREMA: A Predictive Multi-Task Scheduling Algorithm For Preemptible Neural Processing Units2020
- ParaCrawl: Web-Scale Acquisition of Parallel Corpora2020
- Ebird: Elastic Batch for Improving Responsiveness and Throughput of Deep Learning Services2019