RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generation
Preprint
Abstract
Retrieval-Augmented Generation (RAG) systems have emerged as a promising solution to enhance large language models (LLMs) by integrating external knowledge retrieval with generative capabilities. While significant advancements have been made in improving retrieval accuracy and response quality, a critical challenge remains that the internal knowledge integration and retrieval-generation interactions in RAG workflows are largely opaque. This paper introduces RAGTrace, an interactive evaluation system designed to analyze retrieval and generation dynamics in RAG-based workflows. Informed by a comprehensive literature review and expert interviews, the system supports a multi-level analysis approach, ranging from high-level performance evaluation to fine-grained examination of retrieval relevance, generation fidelity, and cross-component interactions. Unlike conventional evaluation practices that focus on isolated retrieval or generation quality assessments, RAGTrace enables an integrated exploration of retrieval-generation relationships, allowing users to trace knowledge sources and identify potential failure cases. The system's workflow allows users to build, evaluate, and iterate on retrieval processes tailored to their specific domains of interest. The effectiveness of the system is demonstrated through case studies and expert evaluations on real-world RAG applications.Keywords
All Related Versions
- Version 1, 2025-08-08, ArXiv
- Published version: , 1.
This publication has 66 references indexed in Scilit:
- Knowledge graph enhanced retrieval-augmented generation for failure mode and effects analysisJournal of Industrial Information Integration, 2025
- Evaluation of Retrieval-Augmented Generation: A Survey2025
- Enhancing Retrieval Processes for Language Generation with Augmented Queries to Provide Factual Information on SchizophreniaProcedia Computer Science, 2024
- Maximizing RAG efficiency: A comparative analysis of RAG methodsNatural Language Processing, 2024
- A survey on large language models for recommendationWorld Wide Web, 2024
- A Comparative Study of Responses to Retina Questions from Either Experts, Expert-Edited Large Language Models, or Expert-Edited Large Language Models AloneOphthalmology Science, 2024
- Interpreting Black-Box Models: A Review on Explainable Artificial IntelligenceCognitive Computation, 2023
- A survey of visual analytics for Explainable Artificial Intelligence methodsComputers & Graphics, 2022
- Adaptive visualization for exploratory information retrievalInformation Processing & Management, 2013
- BRAQUE: Design of an interface to support user interaction in information retrievalInformation Processing & Management, 1993