Toward Communication-Efficient and Data-Free Collaborative Fine-Tuning Between Small and Large Language Models
- 22 May 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Networking
- Vol. 34, 5333-5348
Abstract
While large language models (LLMs) exhibit impressive general capabilities, their performance on domainspecific tasks often requires fine-tuning with private data that cannot be shared due to privacy constraints. Directly deploying LLMs on resource-constrained clients for local fine-tuning is impractical due to their significant computation and communication costs. In addition, pre-trained LLMs are valuable intellectual property, and model owners are reluctant to distribute full model weights. To address these challenges, we propose CoT-LM, a communication-efficient, computation-light, and data-free framework for collaborative fine-tuning between small (SLMs) and large language models (LLMs). In CoT-LM, clients fine-tune lightweight SLMs locally without uploading models or private data. These SLMs provide task-specific feedback to guide server-side LLM enhancement via an efficient communication protocol that exchanges only lightweight synthetic data and feedback. The framework supports both synchronous and asynchronous collaboration and enables mutual enhancement: the LLM improves its task-specific capabilities, while clients benefit from refined synthetic data or distilled knowledge. Extensive experiments demonstrate that CoT-LM significantly boosts natural language understanding (NLU, up to 18.3% for LLMs and 8.0% for SLMs) and natural language generation (NLG, up to 31.7% for LLMs) performance across diverse tasks while preserving data privacy, model intellectual property, and generalization capabilities, achieving significant reductions in computation and communication overhead.Keywords
Funding Information
- National Natural Science Foundation of China (62402267)
- National Natural Science Foundation of China (62432004)
- National Natural Science Foundation of China (62572496)
- Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (JYB2025XDXM122)
- China Postdoctoral Science Foundation (BX20240177)
- China Postdoctoral Science Foundation (2025M771499)
- Natural Science Foundation of Guangdong Province (2026A1515011265)
- Shenzhen Science and Technology Program (JCYJ20250604175500001)
- Young Elite Scientist Sponsorship Program by China Association for Science and Technology (ZB2025-218)
- Breakthrough Institute
This publication has 39 references indexed in Scilit:
- Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language ModelsIEEE Transactions on Mobile Computing, 2025
- Pushing Large Language Models to the 6G Edge: Vision, Challenges, and OpportunitiesIEEE Communications Magazine, 2025
- Towards Building The Federatedgpt: Federated Instruction Tuning2024
- Large language models in medicineNature Medicine, 2023
- FedPrompt: Communication-Efficient and Privacy-Preserving Prompt Tuning in Federated Learning2023
- A Hierarchical Knowledge Transfer Framework for Heterogeneous Federated Learning2023
- ChatGPT for good? On opportunities and challenges of large language models for educationLearning and Individual Differences, 2023
- Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning2022
- A Secure and Privacy‐Preserving Watermark Based Medical Image Sharing MethodChinese Journal of Electronics, 2020
- A Survey on Emerging Computing Paradigms for Big DataChinese Journal of Electronics, 2017