LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
Preprint
Abstract
Fine-tuning large language models (LLMs) is increasingly costly as models scale to hundreds of billions of parameters, and even parameter-efficient fine-tuning (PEFT) methods like LoRA remain resource-intensive. We introduce LowRA, the first framework to enable LoRA fine-tuning below 2 bits per parameter with minimal performance loss. LowRA optimizes fine-grained quantization - mapping, threshold selection, and precision assignment - while leveraging efficient CUDA kernels for scalable deployment. Extensive evaluations across 4 LLMs and 4 datasets show that LowRA achieves a superior performance-precision trade-off above 2 bits and remains accurate down to 1.15 bits, reducing memory usage by up to 50%. Our results highlight the potential of ultra-low-bit LoRA fine-tuning for resource-constrained environments.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the EdgeProceedings of the AAAI Conference on Artificial Intelligence, 2024
- ApiQ: Finetuning of 2-Bit Quantized Large Language Model2024
- Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks2022
- BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models2022
- UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning2022
- BinaryBERT: Pushing the Limit of BERT Quantization2021
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- Least squares quantization in PCMIEEE Transactions on Information Theory, 1982
- The singular value decomposition: Its computation and some applicationsIEEE Transactions on Automatic Control, 1980
- Quantizing for minimum distortionIEEE Transactions on Information Theory, 1960