DistilSum:
Authors:
Open Access
- 19 October 2020
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 2069-2072
Abstract
A popular choice for extractive summarization is to conceptualize it as sentence-level classification, supervised by binary labels. While the common metric ROUGE prefers to measure the text similarity, instead of the performance of classifier. For example, BERTSUMEXT, the best extractive classifier so far, only achieves a precision of 32.9% at the top 3 extracted sentences ([email protected]) on CNN/DM dataset. It is obvious that current approaches cannot model the complex relationship of sentences exactly with 0/1 targets. In this paper, we introduce DistilSum, which contains teacher mechanism and student model. Teacher mechanism produces high entropy soft targets at a high temperature. Our student model is trained with the same temperature to match these informative soft targets and tested with temperature of 1 to distill for ground-truth labels. Compared with large version of BERTSUMEXT, our experimental result on CNN/DM achieves a substantial improvement of 0.99 ROUGE-L score (text similarity) and 3.95 [email protected] score (performance of classifier). Our source code will be available on Github.Keywords
This publication has 10 references indexed in Scilit:
- Distilling Knowledge Learned in BERT for Text Generation2020
- Heterogeneous Graph Neural Networks for Extractive Document Summarization2020
- Extractive Summarization as Text Matching2020
- Multilingual Neural Machine Translation with Language Clustering2019
- Patient Knowledge Distillation for BERT Model Compression2019
- Text Summarization with Pretrained Encoders2019
- Extractive Summarization with SWAP-NET: Sentences and Words from Alternating Pointer Networks2018
- Reinforced Extractive Summarization with Question-Focused Rewards2018
- Ranking Sentences for Extractive Summarization with Reinforcement Learning2018
- Hierarchical Attention Networks for Document Classification2016