Delayed Bottlenecking: Alleviating Forgetting in Pre-trained Graph Neural Networks
- 12 December 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Knowledge and Data Engineering
- Vol. 37 (3) , 1140-1153
Abstract
Pre-training GNNs to extract transferable knowledge and apply it to downstream tasks has become the de facto standard of graph representation learning. Recent works focused on designing self-supervised pre-training tasks to extract useful and universal transferable knowledge from large-scale unlabeled data. However, they have to face an inevitable question: traditional pre-training strategies that aim at extracting useful information about pre-training tasks, may not extract all useful information about the downstream task. In this paper, we reexamine the pre-training process within traditional pre-training and fine-tuning frameworks from the perspective of Information Bottleneck (IB) and confirm that the forgetting phenomenon in pre-training phase may cause detrimental effects on downstream tasks. Therefore, we propose a novel D elayed B ottlenecking P re-training (DBP) framework which maintains as much as possible mutual information between latent representations and training data during pre-training phase by suppressing the compression operation and delays the compression operation to fine-tuning phase to make sure the compression can be guided with labeled fine-tuning data and downstream tasks. To achieve this, we design two information control objectives that can be directly optimized and further integrate them into the actual model design. Extensive experiments on both chemistry and biology domains demonstrate the effectiveness of DBP. The code is available in https://anonymous.4open.science/r/TKDE-DBP .Keywords
All Related Versions
Funding Information
- Natural Science Foundation of China youth project (62402472)
- Natural Science Foundation of Jiangsu Province of China Youth Project (BK20240461, BK20240460)
- Research Grants Council of the Hong Kong Special Administrative Region, China (11215723)
- National Natural Science Foundation of China (62072427, 12227901)
- Project of Stable Support for Youth Team in Basic Research Field, CAS (YSBR-005)
- Academic Leaders Cultivation Program, USTC
- Key Basic Research Foundation of Shenzhen, China (JCYJ20220818100005011)
This publication has 40 references indexed in Scilit:
- Chemistry-intuitive explanation of graph neural networks for molecular property prediction with substructure maskingNature Communications, 2023
- Multi-dimensional Graph Neural Network for Sequential RecommendationPattern Recognition, 2023
- Graph neural networks for materials science and chemistryCommunications Materials, 2022
- S5CL: Unifying Fully-Supervised, Self-supervised, and Semi-supervised Learning Through Hierarchical Contrastive Learning2022
- Targeted BERT Pre-training and Fine-Tuning Approach for Entity Relation Extraction2021
- Prefrontal-hippocampal interactions supporting the extinction of emotional memories: the retrieval stopping modelNeuropsychopharmacology, 2021
- Enhanced Spiking Neural Network with forgetting phenomenon based on electronic synaptic devicesNeurocomputing, 2020
- Bi-Labeled LDA: Inferring Interest Tags for Non-famous Users in Social NetworkData Science and Engineering, 2019
- ZINC 15 – Ligand Discovery for EveryoneJournal of Chemical Information and Modeling, 2015
- Node Classification in Social Networks2011