Measuring Semantic Coherence of RAG-Generated Abstracts Through Complex Network Metrics
Open Access
- 31 October 2025
- journal article
- research article
- Published by MDPI AG in Mathematics
- Vol. 13 (21) , 3472
Abstract
The exponential growth of scientific literature demands scalable methods to evaluate large-language-model outputs beyond surface-level fluency. We present a two-phase framework that separates generation from evaluation: a retrieval-augmented generation system first produces candidate abstracts, which are then embedded into semantic co-occurrence graphs and assessed using seven robustness metrics from complex network theory. Two experiments were conducted. The first varied model, embedding and prompt configurations, achieved results showing clear differences in performance; the best family combined gemma-2b-it, a prompt inspired by chain-of-Thought reasoning, and all-mpnet-base-v2, achieving the highest graph-based robustness. The second experiment refined the temperature setting for this family, identifying as optimal, which stabilized results (sd ) and improved robustness relative to retrieval baselines (, ). While human evaluation was limited to a small set of abstracts, the results revealed a partial convergence between graph-based robustness and expert judgments of coherence and importance. Our approach contrasts with methods like GraphRAG and establishes a reproducible, model-agnostic pathway for the scalable quality control of LLM-generated scientific content.
Keywords
Funding Information
- Agenția Națională pentru Cercetare și Dezvoltare (1231283)
- ANID Doctorado Nacional 2024 (21240115)
This publication has 60 references indexed in Scilit:
- Graph neural networks for text classification: a surveyArtificial Intelligence Review, 2024
- Measuring the novelty of scientific publications: A fastText and local outlier factor approachJournal of Informetrics, 2023
- A method for constructing word sense embeddings based on word sense inductionScientific Reports, 2023
- Using virtual edges to improve the discriminability of co-occurrence text networksPhysica A: Statistical Mechanics and its Applications, 2021
- Assessment of Text Coherence by Constructing the Graph of Semantic, Lexical, and Grammatical Consistancy of Phrases of SentencesCybernetics and Systems Analysis, 2020
- Multi-criteria robustness analysis of metro networksPhysica A: Statistical Mechanics and its Applications, 2017
- Language clustering with word co-occurrence networks based on parallel textsChinese Science Bulletin, 2013
- Cohen’s linearly weighted kappa is a weighted averageAdvances in Data Analysis and Classification, 2011
- Semantic networks based on titles of scientific papersPhysica A: Statistical Mechanics and its Applications, 2011
- Semantic networksComputers & Mathematics with Applications, 1992