Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings
Authors:
Preprint
Abstract
The use of positional embeddings in transformer language models is widely accepted. However, recent research has called into question the necessity of such embeddings. We further extend this inquiry by demonstrating that a randomly initialized and frozen transformer language model, devoid of positional embeddings, inherently encodes strong positional information through the shrinkage of self-attention variance. To quantify this variance, we derive the underlying distribution of each step within a transformer layer. Through empirical validation using a fully pretrained model, we show that the variance shrinkage effect still persists after extensive gradient updates. Our findings serve to justify the decision to discard positional embeddings and thus facilitate more efficient pretraining of transformer language models.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Transformer Language Models without Positional Encodings Still Learn Positional Information2022
- Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little2021
- Positional Artefacts Propagate Through Masked Language Model Embeddings2021
- On the Computational Power of Transformers and Its Implications in Sequence Modeling2020
- What’s so special about BERT’s layers? A closer look at the NLP pipeline in monolingual and multilingual models2020
- Probing Pretrained Language Models for Lexical Semantics2020
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Transformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel2019
- Assessing the Ability of Self-Attention Networks to Learn Word Order2019
- Serial Order: A Parallel Distributed Processing Approach1997