Enhancing Prompt Tuning for Smaller Pretrained Models via Knowledge Distillation
- 24 June 2025
- book chapter
- Published by Springer Nature
- p. 164-178
Abstract
No abstract availableKeywords
This publication has 16 references indexed in Scilit:
- Student-friendly knowledge distillationKnowledge-Based Systems, 2024
- TinyViT: Fast Pretraining Distillation for Small Vision Transformers2022
- BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models2022
- P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks2022
- The Power of Scale for Parameter-Efficient Prompt Tuning2021
- MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers2021
- TinyBERT: Distilling BERT for Natural Language Understanding2020
- Transformers: State-of-the-Art Natural Language Processing2020
- BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingarXiv, 2019
- Distilling the Knowledge in a Neural NetworkarXiv, 2015