A Flexible Multi-Task Model for BERT Serving
Authors:
- Tianwen WeiTWTianwen Wei
- Jianwei QiJQJianwei Qi
- Shenghuan HeSHShenghuan He
Preprint
Abstract
In this demonstration, we present an efficient BERT-based multi-task (MT) framework that is particularly suitable for iterative and incremental development of the tasks. The proposed framework is based on the idea of partial fine-tuning, i.e. only fine-tune some top layers of BERT while keep the other layers frozen. For each task, we train independently a single-task (ST) model using partial fine-tuning. Then we compress the task-specific layers in each ST model using knowledge distillation. Those compressed ST models are finally merged into one MT model so that the frozen layers of the former are shared across the tasks. We exemplify our approach on eight GLUE tasks, demonstrating that it is able to achieve both strong performance and efficiency. We have implemented our method in the utterance understanding system of XiaoAI, a commercial AI assistant developed by Xiaomi. We estimate that our model reduces the overall serving cost by 86%.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- What Happens To BERT Embeddings During Fine-tuning?2020
- BERT-of-Theseus: Compressing BERT by Progressive Module Replacing2020
- AdapterHub: A Framework for Adapting Transformers2020
- TinyBERT: Distilling BERT for Natural Language Understanding2020
- The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding2020
- Patient Knowledge Distillation for BERT Model Compression2019
- Deep Contextualized Word Representations2018
- A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference2018
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016
- Multitask LearningMachine Learning, 1997