Feature Structure Distillation with Centered Kernel Alignment in BERT Transferring
Preprint
Abstract
Knowledge distillation is an approach to transfer information on representations from a teacher to a student by reducing their difference. A challenge of this approach is to reduce the flexibility of the student's representations inducing inaccurate learning of the teacher's knowledge. To resolve it in transferring, we investigate distillation of structures of representations specified to three types: intra-feature, local inter-feature, global inter-feature structures. To transfer them, we introduce feature structure distillation methods based on the Centered Kernel Alignment, which assigns a consistent value to similar features structures and reveals more informative relations. In particular, a memory-augmented transfer method with clustering is implemented for the global structures. The methods are empirically analyzed on the nine tasks for language understanding of the GLUE dataset with Bidirectional Encoder Representations from Transformers (BERT), which is a representative neural language model. In the results, the proposed methods effectively transfer the three types of structures and improve performance compared to state-of-the-art distillation methods. Indeed, the code for the methods is available in https://github.com/maroo-sky/FSD.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression2021
- Distilling Linguistic Context for Language Model Compression2021
- Leveraging Undiagnosed Data for Glaucoma Classification with Teacher-Student Learning2020
- TinyBERT: Distilling BERT for Natural Language Understanding2020
- MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices2020
- Neural Network Acceptability JudgmentsTransactions of the Association for Computational Linguistics, 2019
- Correlation Congruence for Knowledge Distillation2019
- Knowledge Distillation via Instance Relationship Graph2019
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016
- Relations Between Two Sets of Variates1992