Token Sugar: Making Source Code Sweeter for LLMs through Token-Efficient Shorthand
Preprint
Abstract
Large language models (LLMs) have shown exceptional performance in code generation and understanding tasks, yet their high computational costs hinder broader adoption. One important factor is the inherent verbosity of programming languages, such as unnecessary formatting elements and lengthy boilerplate code. This leads to inflated token counts in both input and generated outputs, which increases inference costs and slows down the generation process. Prior work improves this through simplifying programming language grammar, reducing token usage across both code understanding and generation tasks. However, it is confined to syntactic transformations, leaving significant opportunities for token reduction unrealized at the semantic level. In this work, we propose Token Sugar, a concept that replaces frequent and verbose code patterns with reversible, token-efficient shorthand in the source code. To realize this concept in practice, we designed a systematic solution that mines high-frequency, token-heavy patterns from a code corpus, maps each to a unique shorthand, and integrates them into LLM pretraining via code transformation. With this solution, we obtain 799 (code pattern, shorthand) pairs, which can reduce up to 15.1% token count in the source code and is complementary to existing syntax-focused methods. We further trained three widely used LLMs on Token Sugar-augmented data. Experimental results show that these models not only achieve significant token savings (up to 11.2% reduction) during generation but also maintain near-identical Pass@1 scores compared to baselines trained on unprocessed code.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- An Empirical Study of Code Simplification Methods in Code Intelligence TasksACM Transactions on Software Engineering and Methodology, 2025
- LEANCODE: Understanding Models Better for Code Simplification of Pre-trained Large Language Models2025
- Large Language Models for Software Engineering: A Systematic Literature ReviewACM Transactions on Software Engineering and Methodology, 2024
- Natural Is the Best: Model-Agnostic Code Simplification for Pre-trained Large Language ModelsProceedings of the ACM on Software Engineering, 2024
- Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale2024
- babble: Learning Better Abstractions with E-Graphs and Anti-unificationProceedings of the ACM on Programming Languages, 2023
- Program Translation via Code Distillation2023
- Learning Programmatic Idioms for Scalable Semantic Parsing2019
- Mining Semantic Loop IdiomsIEEE Transactions on Software Engineering, 2018
- Interval scheduling: A surveyNaval Research Logistics (NRL), 2007