MENTOR: An Evidence-Informed Large Language Model-Based Multi-Agent System for Efficient Selection of Institution-Specific Negative Control Outcomes
Preprint
Abstract
Objective: Negative control outcomes (NCOs) help detect and correct systematic error in observational studies, yet their selection remains largely manual. The Common Evidence Model (CEM) can generate candidate lists, but researchers must still screen them for plausibility as true nulls and for sufficient observation in each institution’s data. We developed and evaluated MENTOR (Multi-agent Evidence-informed Negative conTrol Outcome selection with Reasoning), a multi-agent system of large language models (LLMs) that combines CEM evidence with LLM reasoning to select institution-specific NCOs. Methods: Four LLM agents operate on an institution’s Observational Medical Outcomes Partnership common data model. A Curator summarizes drug labels, a Suggester assesses locally frequent candidates with CEM evidence and a first-pass LLM judgment, an Adjudicator resolves their disagreements, and a Diagnostician relaxes the patient-count threshold until a target number of NCOs is met. Using Qwen3.6-27B, we evaluated 349 candidates for three drug pairs in a Korean tertiary hospital’s electronic health records (2.9 million patients) against a blinded LLM judge from a different model family, validated by three experts on 90 candidates. We compared accuracy, positive predictive value (PPV), and sensitivity with CEM-only and first-pass-LLM-only selection. Results: MENTOR proposed 243 NCOs versus 121 by CEM-only, with higher accuracy (0.83 [0.79–0.87] vs. 0.53 [0.48–0.58]), sensitivity (0.89 vs. 0.41), and PPV (0.86 vs. 0.79). Judge–expert agreement was κ=0.63. Adjudication, needed for 44.1% of candidates, added little accuracy, and only with reasoning mode. Concordant acceptances, which bypass it, contained about half of the false positives. Only 7%–9% of CEM-proposed outcomes had at least 10,000 patients at the institution. Conclusion: Combining LLM judgments with CEM evidence and local data yielded more NCOs, with higher PPV, than CEM alone. Restricting reasoning to discordant candidates kept computation time below 40% of that with reasoning in all agents. Every decision comes with its evidence and rationale for investigators’ final review.This publication has 33 references indexed in Scilit:
- How Negative Controls Are Used in Pharmacoepidemiology: A Methodological Scoping Review of Real‐World Observational Studies of Glucagon‐Like Peptide‐1 Receptor AgonistsPharmacoepidemiology and Drug Safety, 2026
- AgentMD: Empowering language agents for risk prediction with large-scale clinical tool learningNature Communications, 2025
- Leveraging FDA Labeling Documents and Large Language Model to Enhance Annotation, Profiling, and Classification of Drug Adverse Events with AskFDALabelDrug Safety, 2025
- The TRIPOD-LLM reporting guideline for studies using large language modelsNature Medicine, 2025
- Advances in methodologies of negative controls: a scoping reviewJournal of Clinical Epidemiology, 2023
- Large language models in medicineNature Medicine, 2023
- Comprehensive comparative effectiveness and safety of first-line antihypertensive drug classes: a systematic, multinational, large-scale analysisThe Lancet, 2019
- Empirical confidence interval calibration for population-level effect estimation studies in observational healthcare dataProceedings of the National Academy of Sciences, 2018
- Accuracy of an automated knowledge base for identifying drug adverse reactionsJournal of Biomedical Informatics, 2017
- Interpreting observational studies: why empirical calibration is needed to correct p‐valuesStatistics in Medicine, 2013