CTPD: Cross Tokenizer Preference Distillation
Preprint
Abstract
While knowledge distillation has seen widespread use in pre-training and instruction tuning, its application to aligning language models with human preferences remains underexplored, particularly in the more realistic cross-tokenizer setting. The incompatibility of tokenization schemes between teacher and student models has largely prevented fine-grained, white-box distillation of preference information. To address this gap, we propose Cross-Tokenizer Preference Distillation (CTPD), the first unified framework for transferring human-aligned behavior between models with heterogeneous tokenizers. CTPD introduces three key innovations: (1) Aligned Span Projection, which maps teacher and student tokens to shared character-level spans for precise supervision transfer; (2) a cross-tokenizer adaptation of Token-level Importance Sampling (TIS-DPO) for improved credit assignment; and (3) a Teacher-Anchored Reference, allowing the student to directly leverage the teacher's preferences in a DPO-style objective. Our theoretical analysis grounds CTPD in importance sampling, and experiments across multiple benchmarks confirm its effectiveness, with significant performance gains over existing methods. These results establish CTPD as a practical and general solution for preference distillation across diverse tokenization schemes, opening the door to more accessible and efficient alignment of language models.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model2023
- Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes2023
- Training Language Models to Follow Instructions with Human Feedback2022
- TruthfulQA: Measuring How Models Mimic Human Falsehoods2022
- WinoGrande: An Adversarial Winograd Schema Challenge at ScaleProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Bayesian Estimates of Equation System Parameters: An Application of Integration by Monte CarloEconometrica, 1978