Token-level Ensembling of Models with Different Vocabularies
Preprint
Abstract
Model ensembling is a technique to combine the predicted distributions of two or more models, often leading to improved robustness and performance. For ensembling in text generation, the next token's probability distribution is derived from a weighted sum of the distributions of each individual model. This requires the underlying models to share the same subword vocabulary, limiting the applicability of ensembling, since many open-sourced models have distinct vocabularies. In research settings, experimentation or upgrades to vocabularies may introduce multiple vocabulary sizes. This paper proposes an inference-time only algorithm that allows for ensembling models with different vocabularies, without the need to learn additional parameters or alter the underlying models. Instead, the algorithm ensures that tokens generated by the ensembled models \textit{agree} in their surface form. We apply this technique to combinations of traditional encoder-decoder models and decoder-only LLMs and evaluate on machine translation. In addition to expanding to model pairs that were previously incapable of token-level ensembling, our algorithm frequently improves translation performance over either model individually.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- An Empirical Study of Translation Hypothesis Ensembling with Large Language Models2023
- LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion2023
- XLEnt: Mining a Large Cross-lingual Entity Dataset with Lexical-Semantic-Phonetic Word Alignment2021
- WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia2021
- CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web2021
- Many-to-English Machine Translation Tools, Data, and Pretrained Models2021
- The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT2020
- CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs2020
- Hierarchical Phrase-Based TranslationComputational Linguistics, 2007
- Neural network ensemblesIEEE Transactions on Pattern Analysis and Machine Intelligence, 1990