GiGaMAE: Generalizable Graph Masked Autoencoder via Collaborative Latent Space Reconstruction
Open Access
- 21 October 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 2259-2269
Abstract
Self-supervised learning with masked autoencoders has recently gained popularity for its ability to produce effective image or textual representations, which can be applied to various downstream tasks without retraining. However, we observe that the current masked autoencoder models lack good generalization ability on graph data. To tackle this issue, we propose a novel graph masked autoencoder framework called GiGaMAE. Different from existing masked autoencoders that learn node presentations by explicitly reconstructing the original graph components (e.g., features or edges), in this paper, we propose to collaboratively reconstruct informative and integrated latent embeddings. By considering embeddings encompassing graph topology and attribute information as reconstruction targets, our model could capture more generalized and comprehensive knowledge. Furthermore, we introduce a mutual information based reconstruction loss that enables the effective reconstruction of multiple targets. This learning objective allows us to differentiate between the exclusive knowledge learned from a single target and common knowledge shared by multiple targets. We evaluate our method on three downstream tasks with seven datasets as benchmarks. Extensive experiments demonstrate the superiority of GiGaMAE against state-of-the-art baselines. We hope our results will shed light on the design of foundation models on graph-structured data. Our code is available at: https://github.com/sycny/GiGaMAE.Keywords
All Related Versions
Funding Information
- National Science Foundation (2223768)
This publication has 25 references indexed in Scilit:
- Symmetric Graph Convolutional Autoencoder for Unsupervised Graph Representation Learning2019
- Graph Neural Networks for Social Recommendation2019
- MGAE2017
- node2vec2016
- Auto-encoder based dimensionality reductionNeurocomputing, 2016
- DeepWalk2014
- Distributed large-scale natural graph factorization2013
- Poisson Noise Reduction with Non-local PCAJournal of Mathematical Imaging and Vision, 2013
- Robust principal component analysis?Journal of the ACM, 2011
- Extracting and composing robust features with denoising autoencoders2008