SANTM: Efficient Self-attention-driven Network for Text Matching
Open Access
- 29 November 2021
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Internet Technology
- Vol. 22 (3) , 1-21
Abstract
Self-attention mechanisms have recently been embraced for a broad range of text-matching applications. Self-attention model takes only one sentence as an input with no extra information, i.e., one can utilize the final hidden state or pooling. However, text-matching problems can be interpreted either in symmetrical or asymmetrical scopes. For instance, paraphrase detection is an asymmetrical task, while textual entailment classification and question-answer matching are considered asymmetrical tasks. In this article, we leverage attractive properties of self-attention mechanism and proposes an attention-based network that incorporates three key components for inter-sequence attention: global pointwise features, preceding attentive features, and contextual features while updating the rest of the components. Our model follows evaluation on two benchmark datasets cover tasks of textual entailment and question-answer matching. The proposed efficient Self-attention-driven Network for Text Matching outperforms the state of the art on the Stanford Natural Language Inference and WikiQA datasets with much fewer parameters.Keywords
Funding Information
- European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie (721321)
- Soonchunhyang University Research Fund
- Research Council of Finland (336033, 315896)
- Business Finland (884/31/2018)
- EU H2020 (101016775)
This publication has 37 references indexed in Scilit:
- Deep Residual Learning for Image Recognition2016
- Long Short-Term Memory-Networks for Machine Reading2016
- Key-Value Memory Networks for Directly Reading Documents2016
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016
- Natural Language Inference by Tree-Based Convolution and Heuristic Matching2016
- Pairwise Word Interaction Modeling with Deep Neural Networks for Semantic Similarity Measurement2016
- Learning Natural Language Inference with LSTM2016
- Learning to Rank Short Text Pairs with Convolutional Deep Neural Networks2015
- WikiQA: A Challenge Dataset for Open-Domain Question Answering2015
- A large annotated corpus for learning natural language inference2015