ConveRT: Efficient and Accurate Conversational Representations from Transformers
Authors:
- Matthew HendersonMHMatthew Henderson
- Iñigo CasanuevaICIñigo Casanueva
Preprint
Abstract
General-purpose pretrained sentence encoders such as BERT are not ideal for real-world conversational AI applications; they are computationally heavy, slow, and expensive to train. We propose ConveRT (Conversational Representations from Transformers), a faster, more compact dual sentence encoder specifically optimized for dialog tasks. We pretrain using a retrieval-based response selection task, effectively leveraging quantization and subword-level parameterization in the dual encoder to build a lightweight memory- and energy-efficient model. In our evaluation, we show that ConveRT achieves state-of-the-art performance across widely established response selection tasks. We also demonstrate that the use of extended dialog history as context yields further performance gains. Finally, we show that pretrained representations from the proposed encoder can be transferred to the intent classification task, yielding strong results across three diverse data sets. ConveRT trains substantially faster than standard sentence encoders or previous state-of-the-art dual encoders. With its reduced size and superior performance, we believe this model promises wider portability and scalability for Conversational AI applications.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Q8BERT: Quantized 8Bit BERT2019
- DSTC7 Task 1: Noetic End-to-End Response Selection2019
- Spoken Conversational AI in Video Games2018
- Conversational agents in healthcare: a systematic reviewJournal of the American Medical Informatics Association, 2018
- Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-Based Chatbots2017
- Modeling Ambiguity, Subjectivity, and Diverging Viewpoints in Opinion Question Answering Systems2016
- Smart Reply2016
- Learning to Respond with Deep Neural Networks for Retrieval-Based Human-Computer Conversation System2016
- Rethinking the Inception Architecture for Computer Vision2016