MLaGA: Multimodal Large Language and Graph Assistant
Authors:
- Dongzhe FanDFDongzhe Fan
- Yi FangYFYi Fang
- Jiajin LiuJLJiajin Liu
- Djellel DifallahDDDjellel Difallah
- Qiaoyu TanQTQiaoyu Tan
Preprint
Abstract
Large Language Models (LLMs) have demonstrated substantial efficacy in advancing graph-structured data analysis. Prevailing LLM-based graph methods excel in adapting LLMs to text-rich graphs, wherein node attributes are text descriptions. However, their applications to multimodal graphs--where nodes are associated with diverse attribute types, such as texts and images--remain underexplored, despite their ubiquity in real-world scenarios. To bridge the gap, we introduce the Multimodal Large Language and Graph Assistant (MLaGA), an innovative model that adeptly extends LLM capabilities to facilitate reasoning over complex graph structures and multimodal attributes. We first design a structure-aware multimodal encoder to align textual and visual attributes within a unified space through a joint graph pre-training objective. Subsequently, we implement a multimodal instruction-tuning approach to seamlessly integrate multimodal features and graph structures into the LLM through lightweight projectors. Extensive experiments across multiple datasets demonstrate the effectiveness of MLaGA compared to leading baseline methods, achieving superior performance in diverse graph learning tasks under both supervised and transfer learning scenarios.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- UniGLM: Training One Unified Language Model for Text-Attributed Graphs Embedding2025
- Large Language Models on Graphs: A Comprehensive SurveyIEEE Transactions on Knowledge and Data Engineering, 2024
- GAugLLM: Improving Graph Contrastive Learning for Text-Attributed Graphs with Large Language Models2024
- GraphGPT: Graph Instruction Tuning for Large Language Models2024
- Can we Soft Prompt LLMs for Graph Learning Tasks?2024
- Exploring the Potential of Large Language Models (LLMs)in Learning on GraphsACM SIGKDD Explorations Newsletter, 2024
- Art and the science of generative AIScience, 2023
- Graph Neural Networks in Recommender Systems: A SurveyACM Computing Surveys, 2022
- Graph neural networks: A review of methods and applicationsAI Open, 2020
- The perceptron: A probabilistic model for information storage and organization in the brain.Psychological Review, 1958