On the Effect of Token Merging on Pre-trained Models for Code
Preprint
Abstract
Tokenization is a fundamental component of language models for code. It involves breaking down the input into units that are later passed to the language model stack to learn high-dimensional representations used in various contexts, from classification to generation. However, the output of these tokenizers is often longer than that traditionally used in compilers and interpreters. This could result in undesirable effects, such as increased computational overhead. In this work, we investigate the effect of merging the hidden representations of subtokens that belong to the same semantic unit, such as subtokens that form a single identifier. We propose two strategies: one based on averaging the representations and another that leverages a learning-based approach. Both methods can be seamlessly integrated with existing language models for code. We conduct experiments using six language models for code: CodeBERT, GraphCodeBERT, UniXCoder, CdoeT5, CodeT5+ (220M), and CodeT5+ (770M), across three software engineering tasks: vulnerability detection, code classification, and code translation. Results show that these strategies can reduce the number of floating-point operations by $1\%$ to $19\%$. Regarding downstream performance, the most significant degradation was observed in the vulnerability detection task, where the F1 score decreased by $1.82$ points compared to the baseline. In contrast, for code translation, we observed an improvement of $2.47$ points in CodeBLEU. This work contributes to the broader effort of improving language models for code across multiple dimensions, including both computational efficiency and downstream performance.
Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- An Adaptive Language-Agnostic Pruning Method for Greener Language Models for CodeProceedings of the ACM on Software Engineering, 2025
- Coding-PTMs: How to Find Optimal Code Pre-trained Models for Code Embedding in Vulnerability Detection?2024
- Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMsProceedings of the ACM on Programming Languages, 2024
- Machine Learning is All You Need: A Simple Token-based Approach for Effective Code Clone Detection2024
- Multi-Objective ArchivingIEEE Transactions on Evolutionary Computation, 2023
- An extensive study on pre-trained models for program understanding and generation2022
- Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks2021
- A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries2020
- Big code != big vocabulary2020
- Concise and consistent namingSoftware Quality Journal, 2006