Spelling-out is not Straightforward: LLMs’ Capability of Tokenization from Token to Characters
Open Access
- 1 January 2025
- conference paper
- Published by Association for Computational Linguistics (ACL)
- p. 13340-13353
Abstract
Tatsuya Hiraoka, Kentaro Inui. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- CUTE: Measuring LLMs’ Understanding of Their Tokens2024
- Tokenization Falling Short: On Subword Robustness in Large Language Models2024
- Subword Segmentation in LLMs: Looking at Inflection and Consistency2024
- Neurons in Large Language Models: Dead, N-gram, Positional2024
- Language Models Implement Simple Word2Vec-style Vector Arithmetic2024
- Probing Classifiers: Promises, Shortcomings, and AdvancesComputational Linguistics, 2022
- Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens2022
- What do tokens know about their characters and how do they know it?2022
- Attention is Not Only a Weight: Analyzing Transformers with Vector Norms2020
- Transformers: State-of-the-Art Natural Language Processing2020