Structural Knowledge Distillation: Tractably Distilling Information for Structured Predictor
Preprint
Abstract
Knowledge distillation is a critical technique to transfer knowledge between models, typically from a large model (the teacher) to a more fine-grained one (the student). The objective function of knowledge distillation is typically the cross-entropy between the teacher and the student's output distributions. However, for structured prediction problems, the output space is exponential in size; therefore, the cross-entropy objective becomes intractable to compute and optimize directly. In this paper, we derive a factorized form of the knowledge distillation objective for structured prediction, which is tractable for many typical choices of the teacher and student models. In particular, we show the tractability and empirical effectiveness of structural knowledge distillation between sequence labeling and dependency parsing models under four different scenarios: 1) the teacher and student share the same factorization form of the output structure scoring function; 2) the student factorization produces more fine-grained substructures than the teacher factorization; 3) the teacher factorization produces more fine-grained substructures than the student factorization; 4) the factorization forms from the teacher and the student are incompatible.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- AIN: Fast and Accurate Sequence Labeling with Approximate Inference Network2020
- Named Entity Recognition as Dependency Parsing2020
- Unsupervised Cross-lingual Representation Learning at Scale2020
- Efficient Second-Order TreeCRF for Neural Dependency Parsing2020
- XtremeDistil: Multi-stage Distillation for Massive Multilingual Models2020
- Distilling Neural Networks for Greener and Faster Dependency Parsing2020
- Don’t Eclipse Your Arts Due to Small Discrepancies: Boundary Repositioning with a Pointer Network for Aspect Extraction2020
- Structure-Level Knowledge Distillation For Multilingual Sequence Labeling2020
- Enriching Word Vectors with Subword InformationTransactions of the Association for Computational Linguistics, 2017
- Model compression2006