Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training
Authors:
- Shengrui LiSLShengrui Li
- Fei ZhaoFZFei Zhao
- Kaiyan ZhaoKZKaiyan Zhao
- Jieying YeJYJieying Ye
- Haifeng LiuHLHaifeng Liu
- Fangcheng ShiFSFangcheng Shi
- Zheyong XieZXZheyong Xie
- Yao HuYHYao Hu
- Shaosheng CaoSCShaosheng Cao
Preprint
Abstract
Determining an effective data mixture is a key factor in Large Language Model (LLM) pre-training, where models must balance general competence with proficiency on hard tasks such as math and code. However, identifying an optimal mixture remains an open challenge, as existing approaches either rely on unreliable tiny-scale proxy experiments or require prohibitively expensive large-scale exploration. To address this, we propose Decouple Searching from Training Mix (DeMix), a novel framework that leverages model merging to predict optimal data ratios. Instead of training proxy models for every sampled mixture, DeMix trains component models on candidate datasets at scale and derives data mixture proxies via weighted model merging. This paradigm decouples search from training costs, enabling evaluation of unlimited sampled mixtures without extra training burden and thus facilitating better mixture discovery through more search trials. Extensive experiments demonstrate that DeMix breaks the trade-off between sufficiency, accuracy and efficiency, obtaining the optimal mixture with higher benchmark performance at lower search cost. Additionally, we release the DeMix Corpora, a comprehensive 22T-token dataset comprising high-quality pre-training data with validated mixtures to facilitate open research. Our code and DeMix Corpora is available at https://github.com/Lucius-lsr/DeMix.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications, and OpportunitiesACM Computing Surveys, 2026
- FineMath: A Fine-Grained Mathematical Evaluation Benchmark for Chinese Large Language ModelsACM Transactions on Asian and Low-Resource Language Information Processing, 2025
- Efficient Model Development through Fine-tuning Transfer2025
- Arcee’s MergeKit: A Toolkit for Merging Large Language Models2024
- TIES-Merging: Resolving Interference When Merging Models2023
- Knowledge is a Region in Weight Space for Fine-tuned Language Models2023
- Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing2022
- WinoGrande: An Adversarial Winograd Schema Challenge at ScaleProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Social IQa: Commonsense Reasoning about Social Interactions2019
- ELI5: Long Form Question Answering2019