Abstract
Retrieval-Augmented Generation (RAG) has been widely utilized to integrate external knowledge into Large Language Models (LLMs) for enhancement onmultiple tasks. Recently, graph-guided RAG approaches have demonstrated stronger support for more accurate, context-aware responses. However, traditional RAG methods rely on static pipelines, failing to adaptively balance heterogeneous modalities on complex tasks. To address this issue, we propose GMS-RAG, a novel agentic RAG framework. GMS-RAG operates via a bi-stage autonomous agent system: a Graph-Memory Construction Agent and a Modality-Stratified Retrieval Agent. The construction stage first integrates visual contents along with textual ones into a unified graph memory, then a balanced retrieval mechanism is employed to aggregate evidence for answer generation. Experiments on four benchmarks indicate that GMS-RAG effectively enhances the multi-modal RAG capability of backbone models. Performance measured by F1 score shows that our framework outperforms all baselines with improved quality of answer generation and generalizability over modalities. The code will be available soon.