Sticking to the Mean: Detecting Sticky Tokens in Text Embedding Models
Preprint
Abstract
Despite the widespread use of Transformer-based text embedding models in NLP tasks, surprising 'sticky tokens' can undermine the reliability of embeddings. These tokens, when repeatedly inserted into sentences, pull sentence similarity toward a certain value, disrupting the normal distribution of embedding distances and degrading downstream performance. In this paper, we systematically investigate such anomalous tokens, formally defining them and introducing an efficient detection method, Sticky Token Detector (STD), based on sentence and token filtering. Applying STD to 40 checkpoints across 14 model families, we discover a total of 868 sticky tokens. Our analysis reveals that these tokens often originate from special or unused entries in the vocabulary, as well as fragmented subwords from multilingual corpora. Notably, their presence does not strictly correlate with model size or vocabulary size. We further evaluate how sticky tokens affect downstream tasks like clustering and retrieval, observing significant performance drops of up to 50%. Through attention-layer analysis, we show that sticky tokens disproportionately dominate the model's internal representations, raising concerns about tokenization robustness. Our findings show the need for better tokenization strategies and model design to mitigate the impact of sticky tokens in future text embedding applications.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- GlitchMiner: Mining Glitch Tokens in Large Language Models via Gradient-based Discrete OptimizationProceedings of the AAAI Conference on Artificial Intelligence, 2026
- Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective DetectionProceedings of the ACM on Software Engineering, 2024
- C-Pack: Packed Resources For General Chinese Embeddings2024
- Tokenization Is More Than Compression2024
- Tokenization Falling Short: On Subword Robustness in Large Language Models2024
- Is anisotropy really the cause of BERT embeddings not being semantic?2022
- Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models2022
- SimCSE: Simple Contrastive Learning of Sentence Embeddings2021
- On the Sentence Embeddings from Pre-trained Language Models2020
- Procedures for Detecting Outlying Observations in SamplesTechnometrics, 1969