VERIRAG: Healthcare Claim Verification via Statistical Audit in Retrieval-Augmented Generation
Authors:
- Shubham MoholeSMShubham Mohole
- Hongjun ChoiHCHongjun Choi
- Shusen LiuSLShusen Liu
- Christine KlymkoCKChristine Klymko
- Shashank KushwahaSKShashank Kushwaha
- Derek ShiDSDerek Shi
- Wesam SaklaWSWesam Sakla
- Sainyam GalhotraSGSainyam Galhotra
- Ruben GlattRGRuben Glatt
Preprint
Abstract
Retrieval-augmented generation (RAG) systems are increasingly adopted in clinical decision support, yet they remain methodologically blind-they retrieve evidence but cannot vet its scientific quality. A paper claiming "Antioxidant proteins decreased after alloferon treatment" and a rigorous multi-laboratory replication study will be treated as equally credible, even if the former lacked scientific rigor or was even retracted. To address this challenge, we introduce VERIRAG, a framework that makes three notable contributions: (i) the Veritable, an 11-point checklist that evaluates each source for methodological rigor, including data integrity and statistical validity; (ii) a Hard-to-Vary (HV) Score, a quantitative aggregator that weights evidence by its quality and diversity; and (iii) a Dynamic Acceptance Threshold, which calibrates the required evidence based on how extraordinary a claim is. Across four datasets-comprising retracted, conflicting, comprehensive, and settled science corpora-the VERIRAG approach consistently outperforms all baselines, achieving absolute F1 scores ranging from 0.53 to 0.65, representing a 10 to 14 point improvement over the next-best method in each respective dataset. We will release all materials necessary for reproducing our results.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- The landscape of biomedical researchPatterns, 2024
- The PRISMA 2020 statement: an updated guideline for reporting systematic reviewsBMJ, 2021
- BIOSSES: a semantic sentence similarity estimation system for the biomedical domainBioinformatics, 2017
- The Prevalence of Inappropriate Image Duplication in Biomedical Research PublicationsmBio, 2016
- False-Positive PsychologyPsychological Science, 2011
- The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studiesJournal of Clinical Epidemiology, 2008
- Data Analysis Using Regression and Multilevel/Hierarchical Models2006
- RETRACTED ARTICLE: Elevated Neurotrophin and Neurotrophin Receptor Expression in Spontaneously Hypertensive Rat LungsGrowth Factors, 2004
- Evidence based medicine: what it is and what it isn'tBMJ, 1996
- Inference and missing dataBiometrika, 1976