VIRT: Improving Representation-based Models for Text Matching through Virtual Interaction
Preprint
Abstract
With the booming of pre-trained transformers, representation-based models based on Siamese transformer encoders have become mainstream techniques for efficient text matching. However, these models suffer from severe performance degradation due to the lack of interaction between the text pair, compared with interaction-based models. Prior arts attempt to address this through performing extra interaction for Siamese encoded representations, while the interaction during encoding is still ignored. To remedy this, we propose a \textit{Virtual} InteRacTion mechanism (VIRT) to transfer interactive knowledge from interaction-based models into Siamese encoders through attention map distillation. As a train-time-only component, VIRT could completely maintain the high efficiency of the Siamese structure and brings no extra computation cost during inference. To fully utilize the learned interactive knowledge, we further design a VIRT-adapted interaction strategy. Experimental results on multiple text matching datasets demonstrate that our method outperforms state-of-the-art representation-based models. What's more, VIRT can be easily integrated into existing representation-based methods to achieve further improvements.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- SimCSE: Simple Contrastive Learning of Sentence Embeddings2021
- Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval2021
- ColBERT2020
- TinyBERT: Distilling BERT for Natural Language Understanding2020
- DiPair: Fast and Accurate Distillation for Trillion-Scale Text Matching and Pair Modeling2020
- On the Sentence Embeddings from Pre-trained Language Models2020
- DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering2020
- MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices2020
- Learning to Match using Local and Distributed Representations of Text for Web Search2017
- aNMM2016