Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis
Preprint
Abstract
Length extrapolation permits training a transformer language model on short sequences that preserves perplexities when tested on substantially longer sequences. A relative positional embedding design, ALiBi, has had the widest usage to date. We dissect ALiBi via the lens of receptive field analysis empowered by a novel cumulative normalized gradient tool. The concept of receptive field further allows us to modify the vanilla Sinusoidal positional embedding to create ~\textbf{Sandwich}, the first parameter-free relative positional embedding design that truly length information uses longer than the training sequence. Sandwich shares with KERPLE and T5 the same logarithmic decaying temporal bias pattern with learnable relative positional embeddings; these elucidate future extrapolatable positional embedding design.Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- RoFormer: Enhanced transformer with Rotary Position EmbeddingNeurocomputing, 2023
- ETC: Encoding Long and Structured Inputs in Transformers2020
- Computing Receptive Fields of Convolutional Neural NetworksDistill, 2019
- Deformable Convolutional Networks2017
- Context dependent recurrent neural network language model2012