Reliable extraction of clinical-trial evidence with provenance, verification, and targeted review
Preprint
Abstract
Reliable extraction of trial characteristics and outcomes is essential for systematic reviews, yet incorrect values can alter evidence synthesis and are difficult to detect without clear source evidence. We evaluate EviSearch, a clinical evidence-curation system that links extracted values to source passages, independently verifies cited evidence, and prioritizes higher-risk cells for review. Using ten trial publications, a 133-column schema, reference annotations, and consistent scoring across model families, we assess answer quality, inter-reading agreement, evidence reproduction, targeted review, and evidence modality. In the Qwen evaluation, EviSearch achieved 91.70% accuracy, cited 100.0% of admitted values, and reproduced 98.9% independently. Reviewing 15.6% of cells increased accuracy to 95.22%.Keywords
This publication has 22 references indexed in Scilit:
- Accelerating clinical evidence synthesis with large language modelsnpj Digital Medicine, 2025
- Collaborative large language models for automated data extraction in living systematic reviewsJournal of the American Medical Informatics Association, 2025
- From promise to practice: challenges and pitfalls in the evaluation of large language models for data extraction in evidence synthesisBMJ Evidence-Based Medicine, 2024
- Privacy-preserving large language models for structured medical information retrievalnpj Digital Medicine, 2024
- Performance of two large language models for data extraction in evidence synthesisResearch Synthesis Methods, 2024
- Human-LLM Collaborative Annotation Through Effective Verification of LLM Labels2024
- Can large language models replace humans in systematic reviews? Evaluating GPT‐4's efficacy in screening and extracting data from peer‐reviewed and grey literature in multiple languagesResearch Synthesis Methods, 2024
- AI Chatbots, Health Privacy, and Challenges to HIPAA ComplianceJAMA, 2023
- Validity of data extraction in evidence synthesis practice of adverse events: reproducibility studyBMJ, 2022
- Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registryBMJ Open, 2017