CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Models
Authors:
Open Access
- 1 January 2024
- conference paper
- Published by Association for Computational Linguistics (ACL)
- p. 16143-16162
Abstract
Jiawei Gu, Zacc Yang, Chuanghao Ding, Rui Zhao, Fei Tan. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-TuningIEEE Transactions on Audio, Speech and Language Processing, 2025
- Efficient Domain Continual pretraining by Mitigating the Stability Gap2025
- Continual pre-training mitigates forgetting in language and visionNeural Networks, 2024
- Explaining neural scaling lawsProceedings of the National Academy of Sciences, 2024
- Understanding Emergent Abilities of Language Models from the Loss Perspective2024
- D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models2024
- Balancing Speciality and Versatility: a Coarse to Fine Framework for Supervised Fine-tuning Large Language Model2024
- Continual Training of Language Models for Few-Shot Learning2022
- Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks2020