Benchmarking Large Language Models and Prompt Engineering Strategies in Microsatellite Instability Cancers: Evaluation Study
Open Access
- 21 May 2026
- journal article
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 28, e88614
Abstract
Background: The reliability of general-purpose large language models (LLMs) for complex clinical tasks in specialized domains such as microsatellite instability (MSI) cancers remains critically uncharacterized. The absence of a domain-specific benchmark to evaluate and guide the optimization of their capabilities across diverse clinical tasks poses unevaluated risks to patient safety. Objective: This study aimed to develop and validate Microsatellite Instability Cancer Benchmark (MSIC-Bench), a novel, two-tiered benchmark for MSI cancer, covering both consensus and frontier knowledge. Using this framework, we aimed to systematically assess LLM performance across various prompting strategies, identify task-specific weaknesses, and reveal effective pathways for performance improvement. Methods: We developed MSIC-Bench, a 511-question benchmark derived from clinical guidelines and a curated knowledge base. Three state-of-the-art LLMs (GPT-4o, [OpenAI], Gemini 2.5 Pro [Google], and Claude Opus 4 [Anthropic]) were evaluated using 4 prompting strategies, including vanilla, chain-of-thought, reflection of thoughts, and retrieval-augmented generation (RAG), under both multiple-choice and open-ended modalities. Performance was assessed on accuracy, safety (honesty), error composition, and token usage. Results: LLMs demonstrated a significant “scaffolding effect,” with accuracy dropping substantially in open-ended scenarios. For non-RAG strategies, the primary failure mode was an internal knowledge deficit. The integration of RAG proved to be the most effective intervention. A domain-aligned RAG strategy not only significantly improved accuracy in complex decision-making tasks but also fundamentally shifted the system’s primary bottleneck from knowledge deficits to retrieval failures. In terms of safety, RAG induced a favorable shift from high-risk fabrications to safer refusals, though this introduced a safety-utility trade-off in the form of false refusals. Notably, our hybrid-RAG configuration, which combined both knowledge sources, demonstrated the most robust and generalizable performance across all tasks. Conclusions: Current LLMs lack the specialized knowledge required for reliable application in MSI oncology. A well-designed RAG architecture is the pivotal intervention to address this gap. However, its success is not automatic; it transforms the nature of system failure, making retrieval precision and knowledge base quality the new critical determinants of performance and safety. Our findings establish a clear directive for developing trustworthy clinical artificial intelligence: focus must shift toward optimizing the retrieval component and curating high-quality, comprehensive knowledge sources. MSIC-Bench provides a robust framework to guide these future efforts.Keywords
This publication has 30 references indexed in Scilit:
- NDDRF 2.0: An update and expansion of risk factor knowledge base for personalized prevention of neurodegenerative diseasesAlzheimer's & Dementia, 2025
- Core reference ontology for individualized exercise prescriptionScientific Data, 2024
- Prompt engineering in consistency and reliability with the evidence-based guideline for LLMsnpj Digital Medicine, 2024
- Large language models encode clinical knowledgeNature, 2023
- Artificial intelligence for prediction of response to cancer immunotherapySeminars in Cancer Biology, 2022
- Immune checkpoint inhibitors for the treatment of MSI-H/MMR-D colorectal cancer and a perspective on resistance mechanismsBritish Journal of Cancer, 2019
- Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancerNature Medicine, 2019
- The Cancer Genome Atlas Pan-Cancer analysis projectNature Genetics, 2013
- The Hallmarks of AgingCell, 2013
- Hallmarks of Cancer: The Next GenerationCell, 2011