Why Retrieval-Augmented Generation Fails: A Graph Perspective

Authors:
  • KGKai Guo
    Kai Guo
  • XDXinnan Dai
    Xinnan Dai
  • ZZZhibo Zhang
    Zhibo Zhang
  • NLNuohan Lin
    Nuohan Lin
  • SZShenglai Zeng
    Shenglai Zeng
  • JRJie Ren
    Jie Ren
  • HHHaoyu Han
    Haoyu Han
  • JTJiliang Tang
    Jiliang Tang

Abstract
Retrieval-Augmented Generation (RAG) has become a powerful and widely used approach for improving large language models by grounding generation in retrieved evidence. However, RAG systems still produce incorrect answers in many cases. Why RAG fails despite having access to external information remains poorly understood. We present a model-internal study of retrieval-augmented generation that examines how retrieved evidence influences answer generation. Using circuit tracing, we construct attribution graphs that model the flow of information through transformer layers during decoding. These graphs represent interactions among retrieved context, intermediate model activations, and generated tokens, providing a graph, circuit-level view of how external evidence is integrated into the model's reasoning process across multiple question answering benchmarks, we observe consistent structural differences: correct predictions exhibit deeper reasoning paths, more distributed evidence flow, and a more structured pattern of local connectivity, while failed predictions show shallower, fragmented, and overly concentrated evidence flow. Building on these findings, we develop a graph-based error detection framework that uses attribution-graph topology features. Furthermore, we show that attribution graphs enable targeted interventions. By reinforcing question-constrained evidence grounding, we reshape internal routing so that answer generation remains guided by the question, leading to more effective integration of retrieved information and fewer errors.
All Related Versions