EPAS: Efficient Training with Progressive Activation Sharing
Authors:
Preprint
Abstract
We present a novel method for Efficient training with Progressive Activation Sharing (EPAS). This method bridges progressive training paradigm with the phenomenon of redundant QK (or KV ) activations across deeper layers of transformers. EPAS gradually grows a sharing region during training by switching decoder layers to activation sharing mode. This results in throughput increase due to reduced compute. To utilize deeper layer redundancy, the sharing region starts from the deep end of the model and grows towards the shallow end. The EPAS trained models allow for variable region lengths of activation sharing for different compute budgets during inference. Empirical evaluations with QK activation sharing in LLaMA models ranging from 125M to 7B parameters show up to an 11.1% improvement in training throughput and up to a 29% improvement in inference throughput while maintaining similar loss curve to the baseline models. Furthermore, applying EPAS in continual pretraining to transform TinyLLaMA into an attention-sharing model yields up to a 10% improvement in average accuracy over state-of-the-art methods, emphasizing the significance of progressive training in cross layer activation sharing models.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and ApplicationACM Transactions on Intelligent Systems and Technology, 2025
- A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and RecommendationsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
- LLM in a flash: Efficient Large Language Model Inference with Limited Memory2024
- SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training2024
- LLaMA Pro: Progressive LLaMA with Block Expansion2024
- SparseFlow: Accelerating Transformers by Sparsifying Information Flows2024
- AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Models2024
- Time Sensitive Knowledge Editing through Efficient Finetuning2024
- Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes2023
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness2022