Constructing a BPE tokenization DFA
Authors:
- Martin BerglundMBMartin Berglund
- Willeke MartensWMWilleke Martens
- Brink van der MerweBMBrink van der Merwe
Open Access
- 23 June 2026
- journal article
- Published by Elsevier in Theoretical Computer Science
- Vol. 1083
Abstract
No abstract availableFunding Information
- Knut och Alice Wallenbergs Stiftelse
This publication has 6 references indexed in Scilit:
- Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective DetectionProceedings of the ACM on Software Engineering, 2024
- SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing2018
- Determinization of transducers over finite and infinite wordsTheoretical Computer Science, 2002
- Families of locally testable languagesTheoretical Computer Science, 2000
- Local languages and the Berry-Sethi algorithmTheoretical Computer Science, 1996
- Codes circulaires, automates locaux et entropieTheoretical Computer Science, 1988