A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-Distillation
Authors:
- Lirong WuLWLirong Wu
- Haitao LinHLHaitao Lin
- Zhangyang GaoZGZhangyang Gao
- Guojiang ZhaoGZGuojiang Zhao
- Stan Z. LiSLStan Z. Li
Preprint
Abstract
Recent years have witnessed great success in handling graph-related tasks with Graph Neural Networks (GNNs). Despite their great academic success, Multi-Layer Perceptrons (MLPs) remain the primary workhorse for practical industrial applications. One reason for such an academic-industry gap is the neighborhood-fetching latency incurred by data dependency in GNNs. To reduce their gaps, Graph Knowledge Distillation (GKD) is proposed, usually based on a standard teacher-student architecture, to distill knowledge from a large teacher GNN into a lightweight student GNN or MLP. However, we found in this paper that neither teachers nor GNNs are necessary for graph knowledge distillation. We propose a Teacher-Free Graph Self-Distillation (TGS) framework that does not require any teacher model or GNNs during both training and inference. More importantly, the proposed TGS framework is purely based on MLPs, where structural information is only implicitly used to guide dual knowledge self-distillation between the target node and its neighborhood. As a result, TGS enjoys the benefits of graph topology awareness in training but is free from data dependency in inference. Extensive experiments have shown that the performance of vanilla MLPs can be greatly improved with dual self-distillation, e.g., TGS improves over vanilla MLPs by 15.54% on average and outperforms state-of-the-art GKD algorithms on six real-world datasets. In terms of inference speed, TGS infers 75X-89X faster than existing GNNs and 16X-25X faster than classical inference acceleration methods.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Distilled Neural Networks for Efficient Learning to Rank: (Extended Abstract)2024
- HiSTGNN: Hierarchical spatio-temporal graph neural network for weather forecastingInformation Sciences, 2023
- GNN Cleaner: Label Cleaner for Graph Structured DataIEEE Transactions on Knowledge and Data Engineering, 2023
- OMG: Towards Effective Graph Classification Against Label NoiseIEEE Transactions on Knowledge and Data Engineering, 2023
- Graph neural network for traffic forecasting: A surveyExpert Systems with Applications, 2022
- CCGL: Contrastive Cascade Graph LearningIEEE Transactions on Knowledge and Data Engineering, 2022
- Self-Supervised Learning on Graphs: Contrastive, Generative, or PredictiveIEEE Transactions on Knowledge and Data Engineering, 2021
- Variational Graph Neural Networks for Road Traffic Prediction in Intelligent Transportation SystemsIEEE Transactions on Industrial Informatics, 2020
- Graph neural networks: A review of methods and applicationsAI Open, 2020
- Learning Distilled Graph for Large-Scale Social Network Data ClusteringIEEE Transactions on Knowledge and Data Engineering, 2019