DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models
Authors:
Preprint
Abstract
Low-rank adaptation (LoRA) reduces the computational and memory demands of fine-tuning large language models (LLMs) by approximating updates with low-rank matrices. However, low-rank approximation in two-dimensional space fails to capture high-dimensional structures within the target matrix. Recently, tensor decomposition methods have been explored for fine-tuning LLMs, leveraging their ability to extract structured information. Yet, these approaches primarily rely on random initialization, and the impact of initialization on tensor adaptation remains underexplored. In this paper, we reveal that random initialization significantly diverges from the validation loss achieved by full fine-tuning. To address this, we propose Weight-Decomposed Tensor Adaptation (DoTA), which leverages the Matrix Product Operator (MPO) decomposition of pre-trained weights for effective initialization in fine-tuning LLMs. Additionally, we introduce QDoTA, a quantized version of DoTA designed for 4-bit quantization. Experiments on commonsense and arithmetic reasoning tasks show that DoTA outperforms random initialization methods with fewer parameters. QDoTA further reduces memory consumption and achieves comparable performance to DoTA on commonsense reasoning tasks. We will release our code to support future research.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- EdgeLLM: A Highly Efficient CPU-FPGA Heterogeneous Edge Accelerator for Large Language ModelsIEEE Transactions on Circuits and Systems I: Regular Papers, 2025
- MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning2025
- LoRA-GA: Low-Rank Adaptation with Gradient Approximation2024
- PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models2024
- QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation2024
- LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models2024
- LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models2023
- Compressing deep neural networks by matrix product operatorsPhysical Review Research, 2020
- Matrix product operator representationsNew Journal of Physics, 2010
- Tensor Rank and the Ill-Posedness of the Best Low-Rank Approximation ProblemSIAM Journal on Matrix Analysis and Applications, 2008