RoFormer: Enhanced Transformer with Rotary Position Embedding
Preprint
Abstract
Position encoding recently has shown effective in the transformer architecture. It enables valuable supervision for dependency modeling between elements at different positions of the sequence. In this paper, we first investigate various methods to integrate positional information into the learning process of transformer-based language models. Then, we propose a novel method named Rotary Position Embedding(RoPE) to effectively leverage the positional information. Specifically, the proposed RoPE encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation. Notably, RoPE enables valuable properties, including the flexibility of sequence length, decaying inter-token dependency with increasing relative distances, and the capability of equipping the linear self-attention with relative position encoding. Finally, we evaluate the enhanced transformer with rotary position embedding, also called RoFormer, on various long text classification benchmark datasets. Our experiments show that it consistently overcomes its alternatives. Furthermore, we provide a theoretical analysis to explain some experimental results. RoFormer is already integrated into Huggingface: \url{https://huggingface.co/docs/transformers/model_doc/roformer}.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Improve Transformer Models with Better Relative Position Embeddings2020
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Transformer-XL: Attentive Language Models beyond a Fixed-Length Context2019
- fairseq: A Fast, Extensible Toolkit for Sequence Modeling2019
- A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference2018
- Self-Attention with Relative Position Representations2018
- UdL at SemEval-2017 Task 1: Semantic Textual Similarity Estimation of English Sentence Pairs Using Regression Model over Pairwise Features2017
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016
- Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books2015
- Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank2013