Cross-LoRA: A Data-Free LoRA Transfer Framework across Heterogeneous LLMs
Preprint
Abstract
Traditional parameter-efficient fine-tuning (PEFT) methods such as LoRA are tightly coupled with the base model architecture, which constrains their applicability across heterogeneous pretrained large language models (LLMs). To address this limitation, we introduce Cross-LoRA, a data-free framework for transferring LoRA modules between diverse base models without requiring additional training data. Cross-LoRA consists of two key components: (a) LoRA-Align, which performs subspace alignment between source and target base models through rank-truncated singular value decomposition (SVD) and Frobenius-optimal linear transformation, ensuring compatibility under dimension mismatch; and (b) LoRA-Shift, which applies the aligned subspaces to project source LoRA weight updates into the target model parameter space. Both components are data-free, training-free, and enable lightweight adaptation on a commodity GPU in 20 minutes. Experiments on ARCs, OBOA and HellaSwag show that Cross-LoRA achieves relative gains of up to 5.26% over base models. Across other commonsense reasoning benchmarks, Cross-LoRA maintains performance comparable to that of directly trained LoRA adapters.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning2021
- Revealing the Dark Secrets of BERT2019
- HellaSwag: Can a Machine Really Finish Your Sentence?2019
- Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering2018
- SYMMETRIC GAUGE FUNCTIONS AND UNITARILY INVARIANT NORMSThe Quarterly Journal of Mathematics, 1960
- The Approximation of One Matrix by Another of Lower RankPsychometrika, 1936