Amalgamating Knowledge from Heterogeneous Graph Neural Networks
Authors:
- 1 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 15704-15713
Abstract
In this paper, we study a novel knowledge transfer task in the domain of graph neural networks (GNNs). We strive to train a multi-talented student GNN, without accessing human annotations, that “amalgamates” knowledge from a couple of teacher GNNs with heterogeneous architectures and handling distinct tasks. The student derived in this way is expected to integrate the expertise from both teachers while maintaining a compact architecture. To this end, we propose an innovative approach to train a slimmable GNN that enables learning from teachers with varying feature dimensions. Meanwhile, to explicitly align topological semantics between the student and teachers, we introduce a topological attribution map (TAM) to highlight the structural saliency in a graph, based on which the student imitates the teachers’ ways of aggregating information from neighbors. Experiments on seven datasets across various tasks, including multi-label classification and joint segmentation-classification, demonstrate that the learned student, with a lightweight architecture, achieves gratifying results on par with and sometimes even superior to those of the teachers in their specializations. Our code is publicly available at https://github.com/ycjing/AmalgamateGNN.PyTorch.Keywords
Funding Information
- Australian Research Council
This publication has 33 references indexed in Scilit:
- Online Multi-Task Learning Framework for Ensemble ForecastingIEEE Transactions on Knowledge and Data Engineering, 2017
- A scalable active framework for region annotation in 3D shape collectionsACM Transactions on Graphics, 2016
- A Multi-Task Learning Formulation for Survival Analysis2016
- GSpartan: a Geospatio-Temporal Multi-task Learning Framework for Multi-location Prediction2016
- Novel applications of multitask learning and multiple output regression to multiple genetic trait predictionBioinformatics, 2016
- Multi-task Recurrent Neural Network for Immediacy Prediction2015
- Image-Based Recommendations on Styles and Substitutes2015
- Deep neural networks employing Multi-Task Learning and stacked bottleneck features for speech synthesis2015
- Efficient multi-task feature learning with calibration2014
- Convex multi-task feature learningMachine Learning, 2008