Efficiently Distilling LLMs for Edge Applications
Preprint
Abstract
Supernet training of LLMs is of great interest in industrial applications as it confers the ability to produce a palette of smaller models at constant cost, regardless of the number of models (of different size / latency) produced. We propose a new method called Multistage Low-rank Fine-tuning of Super-transformers (MLFS) for parameter-efficient supernet training. We show that it is possible to obtain high-quality encoder models that are suitable for commercial edge applications, and that while decoder-only models are resistant to a comparable degree of compression, decoders can be effectively sliced for a significant reduction in training time.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- A Survey on Model Compression for Large Language ModelsTransactions of the Association for Computational Linguistics, 2024
- You only compress once: Towards effective and elastic BERT compression via exploit–explore stochastic nature gradientNeurocomputing, 2024
- Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts2024
- Transfer-Once-For-All: AI Model Optimization for Edge2023
- Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes2023
- BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models2022
- TinyBERT: Distilling BERT for Natural Language Understanding2020
- XtremeDistil: Multi-stage Distillation for Massive Multilingual Models2020
- HAT: Hardware-Aware Transformers for Efficient Natural Language Processing2020
- Patient Knowledge Distillation for BERT Model Compression2019