Multimodal learning with graphs
Authors:
Top Cited Papers
- 3 April 2023
- journal article
- research article
- Published by Springer Nature in Nature Machine Intelligence
- Vol. 5 (4) , 340-350
Abstract
Artificial intelligence for graphs has achieved remarkable success in modelling complex systems, ranging from dynamic networks in biology to interacting particle systems in physics. However, the increasingly heterogeneous graph datasets call for multimodal methods that can combine different inductive biases — assumptions that algorithms use to make predictions for inputs they have not encountered during training. Learning on multimodal datasets is challenging because the inductive biases can vary by data modality and graphs might not be explicitly given in the input. To address these challenges, graph artificial intelligence methods combine different modalities while leveraging cross-modal dependencies through geometric relationships. Diverse datasets are combined using graphs and fed into sophisticated multimodal architectures, specified as image-intensive, knowledge-grounded and language-intensive models. Using this categorization, we introduce a blueprint for multimodal graph learning, use it to study existing methods and provide guidelines to design new models.Keywords
All Related Versions
This publication has 85 references indexed in Scilit:
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Deeply-Recursive Convolutional Network for Image Super-Resolution2016
- Fully Convolutional Networks for Semantic SegmentationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Multimodal Deep Autoencoder for Human Pose RecoveryIEEE Transactions on Image Processing, 2015
- Fully convolutional networks for semantic segmentation2015
- SLIC Superpixels Compared to State-of-the-Art Superpixel MethodsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2012
- The Graph Neural Network ModelIEEE Transactions on Neural Networks, 2008
- Reduced surface: An efficient way to compute molecular surfacesBiopolymers, 1996
- Reduced surface: an efficient way to compute molecular surfaces.1996
- Chirality in bioactive agents and its pitfallsTrends in Pharmacological Sciences, 1986