PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning
Preprint
Abstract
Low-rank adaptation (LoRA) and its variants have recently gained much interest due to their ability to avoid excessive inference costs. However, LoRA still encounters the following challenges: (1) Limitation of low-rank assumption; and (2) Its initialization method may be suboptimal. To this end, we propose PMSS(Pre-trained Matrices Skeleton Selection), which enables high-rank updates with low costs while leveraging semantic and linguistic information inherent in pre-trained weight. It achieves this by selecting skeletons from the pre-trained weight matrix and only learning a small matrix instead. Experiments demonstrate that PMSS outperforms LoRA and other fine-tuning methods across tasks with much less trainable parameters. We demonstrate its effectiveness, especially in handling complex tasks such as DROP benchmark(+3.4%/+5.9% on LLaMA2-7B/13B) and math reasoning(+12.89%/+5.61%/+3.11% on LLaMA2-7B, Mistral-7B and Gemma-7B of GSM8K). The code and model will be released soon.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters2025
- MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning2025
- ReFT: Representation Finetuning for Language Models2024
- QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation2024
- CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning2024
- Simpler is better: a comparative study of randomized pivoting algorithms for CUR and interpolative decompositionsAdvances in Computational Mathematics, 2023
- Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Models2023
- Social IQa: Commonsense Reasoning about Social Interactions2019
- Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems2017
- Algorithms on Strings, Trees and Sequences1997