DoRA: Weight-Decomposed Low-Rank Adaptation
Authors:
- Shih-Yang LiuSLShih-Yang Liu
- Chien-Yi WangCWChien-Yi Wang
- Hongxu YinHYHongxu Yin
- Pavlo MolchanovPMPavlo Molchanov
- Yu-Chiang Frank WangYWYu-Chiang Frank Wang
- Kwang-Ting ChengKCKwang-Ting Cheng
- Min-Hung ChenMCMin-Hung Chen
Preprint
Abstract
Among the widely used parameter-efficient finetuning (PEFT) methods, LoRA and its variants have gained considerable popularity because of avoiding additional inference costs. However, there still often exists an accuracy gap between these methods and full fine-tuning (FT). In this work, we first introduce a novel weight decomposition analysis to investigate the inherent differences between FT and LoRA. Aiming to resemble the learning capacity of FT from the findings, we propose Weight-Decomposed LowRank Adaptation (DoRA). DoRA decomposes the pre-trained weight into two components, magnitude and direction, for fine-tuning, specifically employing LoRA for directional updates to efficiently minimize the number of trainable parameters. By employing DoRA, we enhance both the learning capacity and training stability of LoRA while avoiding any additional inference overhead. DoRA consistently outperforms LoRA on fine-tuning LLaMA, LLaVA, and VL-BART on various downstream tasks, such as commonsense reasoning, visual instruction tuning, and image/video-text understanding.Keywords
All Related Versions
This publication has 2 references indexed in Scilit:
- Tied-LoRA: Enhancing parameter efficiency of LoRA with Weight Tying2024
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017