SubTokenTest: A Practical Benchmark for Real-World Sub-token Understanding
Authors:
Preprint
Abstract
Recent advancements in large language models (LLMs) have significantly enhanced their reasoning capabilities. However, they continue to struggle with basic character-level tasks, such as counting letters in words, a problem rooted in their tokenization process. While existing benchmarks have highlighted this weakness through basic character operations, such failures are often dismissed due to lacking practical relevance. Yet, many real-world applications, such as navigating text-based maps or interpreting structured tables, rely heavily on precise sub-token understanding. In this regard, we introduce SubTokenTest, a comprehensive benchmark that assesses sub-token understanding through practical, utility-driven tasks. Our benchmark includes ten tasks across four domains and isolates tokenization-related failures by decoupling performance from complex reasoning. We provide a comprehensive evaluation of nine advanced LLMs. Additionally, we investigate the impact of test-time scaling on sub-token reasoning and explore how character-level information is encoded within the hidden states.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- CharBench: Evaluating the Role of Tokenization in Character-Level TasksProceedings of the AAAI Conference on Artificial Intelligence, 2026
- Spelling-out is not Straightforward: LLMs’ Capability of Tokenization from Token to Characters2025
- Word Form Matters: LLMs’ Semantic Reconstruction under Typoglycemia2025
- CUTE: Measuring LLMs’ Understanding of Their Tokens2024
- Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?2023
- LMentry: A Language Model Benchmark of Elementary Language Tasks2023
- Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them2023
- Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens2022
- BPE-Dropout: Simple and Effective Subword Regularization2020
- Neural Machine Translation of Rare Words with Subword Units2016