Gradient Weight-normalized Low-rank Projection for Efficient LLM Training
Open Access
- 11 April 2025
- journal article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (23) , 24123-24131
Abstract
Large Language Models (LLMs) have shown remarkable performance across various tasks, but the escalating demands on computational resources pose significant challenges, particularly in the extensive utilization of full fine-tuning for downstream tasks. To address this, parameter-efficient fine-tuning (PEFT) methods have been developed, but they often underperform compared to full fine-tuning and struggle with memory efficiency. In this work, we introduce Gradient Weight-Normalized Low-Rank Projection (GradNormLoRP), a novel approach that enhances both parameter and memory efficiency while maintaining comparable performance to full fine-tuning. GradNormLoRP normalizes the weight matrix to improve gradient conditioning, facilitating better convergence during optimization. Additionally, it applies low-rank approximations to the weight and gradient matrices, significantly reducing memory usage during training. Extensive experiments demonstrate that our 8-bit GradNormLoRP reduces optimizer memory usage by up to 89.5\% and enables the pre-training of large LLMs, such as LLaMA 7B, on consumer-level GPUs like the NVIDIA RTX 4090, without additional inference costs. Moreover, GradNormLoRP outperforms existing low-rank methods in fine-tuning tasks. For instance, when fine-tuning the RoBERTa model on all GLUE tasks with a rank of 8, GradNormLoRP achieves an average score of 80.65, surpassing LoRA's score of 79.23. These results underscore GradNormLoRP as a promising alternative for efficient LLM pre-training and fine-tuning.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Full Parameter Fine-tuning for Large Language Models with Limited Resources2024
- $\cal{Y}$-Tuning: an efficient tuning paradigm for large-scale pre-trained models via label representation learningFrontiers of Computer Science, 2023
- LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning2022
- AdapterDrop: On the Efficiency of Adapters in Transformers2021
- AdapterFusion: Non-Destructive Task Composition for Transfer Learning2021
- Prefix-Tuning: Optimizing Continuous Prompts for Generation2021
- WARP: Word-level Adversarial ReProgramming2021
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding2018