Visualization and Analysis of the Loss Landscape in Graph Neural Networks
Authors:
- Samir MoustafaSMSamir Moustafa
- Lorenz KummerLKLorenz Kummer
- Simon FetzelSFSimon Fetzel
- Nils M. KriegeNKNils M. Kriege
- Wilfried N. GanstererWGWilfried N. Gansterer
Preprint
Abstract
Graph Neural Networks (GNNs) are powerful models for graph-structured data, with broad applications. However, the interplay between GNN parameter optimization, expressivity, and generalization remains poorly understood. We address this by introducing an efficient learnable dimensionality reduction method for visualizing GNN loss landscapes, and by analyzing the effects of over-smoothing, jumping knowledge, quantization, sparsification, and preconditioner on GNN optimization. Our learnable projection method surpasses the state-of-the-art PCA-based approach, enabling accurate reconstruction of high-dimensional parameters with lower memory usage. We further show that architecture, sparsification, and optimizer's preconditioning significantly impact the GNN optimization landscape and their training process and final prediction performance. These insights contribute to developing more efficient designs of GNN architectures and training strategies.Keywords
All Related Versions
- Version 1, 2025-09-15, ArXiv
- Published version: , 69.
This publication has 9 references indexed in Scilit:
- Deep Residual Learning for Image Recognition: A SurveyApplied Sciences, 2022
- Graph Attention Multi-Layer Perceptron2022
- A Survey of Quantization Methods for Efficient Neural Network Inference2022
- Optimization of Graph Neural Networks with Natural Gradient Descent2020
- Theoretical Insights Into the Optimization Landscape of Over-Parameterized Shallow Neural NetworksIEEE Transactions on Information Theory, 2018
- Collective Classification in Network DataAI Magazine, 2008
- The elements of statistical learning: data mining, inference and predictionThe Mathematical Intelligencer, 2005
- CiteSeer1998
- Flat MinimaNeural Computation, 1997