Sequence-aware Knowledge Distillation for a Lightweight Event Representation
Open Access
- 21 December 2022
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Information Systems
- Vol. 41 (2) , 1-30
Abstract
Event representation targets to model the event-reasoning process as a machine-readable format. Previous studies on event representation mostly concentrate on a sole modeling perspective and have not well investigated the scenario-level knowledge, which can cause information loss. To cope with this dilemma, we propose a unified fine-tuning architecture-based approach ( UniFA-S ) that integrates all levels of trainings, including the scenario-level knowledge. However, another challenge for existing models is the ever-increasing computation overheads, restricting the deployment ability on limited resources devices. Hence, in this article, we aim to compress the cumbersome model UniFA-S into a lighter and easy-to-deploy one without much performance damage. To this end, we propose a sequence-aware knowledge distillation model (SaKD) that employs a dynamic self-distillation on the decouple-compress-couple framework for compressing UniFA-S , which cannot only realize the model compression, but also retain the integrity of individual components. We also design two fitting strategies to address the less-supervised issue at the distillation stage. Comprehensive experiments on representation-and-inference ability-based tasks validate the effectiveness of SaKD. Compared to UniFA-S , SaKD realizes a more portable event representation model at the cost of only 1.0% performance drop in terms of accuracy or Spearman’s correlation, which is far less than other knowledge distillation models.Keywords
This publication has 35 references indexed in Scilit:
- Learning Efficient Convolutional Networks through Network Slimming2017
- Channel Pruning for Accelerating Very Deep Neural Networks2017
- Densely Connected Convolutional Networks2017
- Integrating Order Information and Event Relation for Script Event Prediction2017
- Deep Residual Learning for Image Recognition2016
- Rethinking the Inception Architecture for Computer Vision2016
- Quantized Convolutional Neural Networks for Mobile Devices2016
- A Study of Entanglement in a Categorical Framework of Natural LanguageElectronic Proceedings in Theoretical Computer Science, 2014
- Scripts, Plans, Goals, and Understanding2013
- Unsupervised learning of narrative schemas and their participants2009