Improving Abstractive Summarization with Unsupervised Dynamic LoRA Mixtures
- 1 July 2024
- journal article
- research article
- Published by Pleiades Publishing Ltd in Lobachevskii Journal of Mathematics
- Vol. 45 (7) , 2995-3006
Abstract
The recent jump in hardware requirements for state-of-the-art language models have increased training and inference costs considerably. While methods such as Mixture-of-Experts (MoE) address the network throughput decline they retain high memory consumption. To alleviate the issue researches explored the possibility of replicating MoE results with layer adapters such as LoRA using various ensembling schemes. However, the utilization of the pre-trained LoRA adapters in the current approaches is limited as they are only considered in static ensembles while the dynamic counterparts are designed to be trained from scratch. We argue that dynamic LoRA ensembles can be obtained without additional fine-tuning by leveraging the network inference statistics. We propose Unsupervised Dynamic LoRA Mixtures (UDLM), a dynamic adapter ensembling method that translates pre-trained LoRA adapter sets to equivalent MoE networks by accumulating adapter spatial statistics. Evaluation on 6 abstractive summarization datasets in knowledge transfer setting demonstrates that UDLM substantially outperforms best static ensembles having a 30\(\%\) summarization quality improvement while introducing only 35\(\%\) increased latency.
Keywords
This publication has 13 references indexed in Scilit:
- AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models2023
- Combining Parameter-efficient Modules for Task-level Generalisation2023
- Parameter-efficient Weight Ensembling Facilitates Task-level Knowledge Transfer2023
- UniSumm and SummZoo: Unified Model and Diverse Benchmark for Few-Shot Summarization2023
- AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning2022
- BOOKSUM: A Collection of Datasets for Long-form Narrative Summarization2022
- TLDR: Extreme Summarization of Scientific Documents2020
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization2019
- Adaptive Mixtures of Local ExpertsNeural Computation, 1991