An Introductory Survey on Attention Mechanisms in NLP Problems
Author:
Preprint
Abstract
First derived from human intuition, later adapted to machine translation for automatic token alignment, attention mechanism, a simple method that can be used for encoding sequence data based on the importance score each element is assigned, has been widely applied to and attained significant improvement in various tasks in natural language processing, including sentiment classification, text summarization, question answering, dependency parsing, etc. In this paper, we survey through recent works and conduct an introductory summary of the attention mechanism in different NLP problems, aiming to provide our readers with basic knowledge on this widely used method, discuss its different variants for different tasks, explore its association with other techniques in machine learning, and examine methods for evaluating its performance.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language UnderstandingProceedings of the AAAI Conference on Artificial Intelligence, 2018
- Multi-Level Structured Self-Attentions for Distantly Supervised Relation Extraction2018
- Self-Attention with Relative Position Representations2018
- Enriching Word Vectors with Subword InformationTransactions of the Association for Computational Linguistics, 2017
- Coupled Multi-Layer Attentions for Co-Extraction of Aspect and Opinion TermsProceedings of the AAAI Conference on Artificial Intelligence, 2017
- A Nested Attention Neural Hybrid Model for Grammatical Error Correction2017
- Hierarchically-Attentive RNN for Album Summarization and Storytelling2017
- Key-Value Memory Networks for Directly Reading Documents2016
- Effective Approaches to Attention-based Neural Machine Translation2015