Comprehensive testing of large language models for extraction of structured data in pathology
Top Cited Papers
Open Access
- 31 March 2025
- journal article
- research article
- Published by Springer Nature in Communications Medicine
- Vol. 5 (1) , 1-13
Abstract
Background: Pathology departments generate large volumes of unstructured data as free-text diagnostic reports. Converting these reports into structured formats for analytics or artificial intelligence projects requires substantial manual effort by specialized personnel. While recent studies show promise in using advanced language models for structuring pathology data, they primarily rely on proprietary models, raising cost and privacy concerns. Additionally, important aspects such as prompt engineering and model quantization for deployment on consumer-grade hardware remain unaddressed. Methods: We created a dataset of 579 annotated pathology reports in German and English versions. Six language models (proprietary: GPT-4; open-source: Llama2 13B, Llama2 70B, Llama3 8B, Llama3 70B, and Qwen2.5 7B) were evaluated for their ability to extract eleven key parameters from these reports. Additionally, we investigated model performance across different prompt engineering strategies and model quantization techniques to assess practical deployment scenarios. Results: Here we show that open-source language models extract structured data from pathology reports with high precision, matching the accuracy of proprietary GPT-4 model. The precision varies significantly across different models and configurations. These variations depend on specific prompt engineering strategies and quantization methods used during model deployment. Conclusions: Open-source language models demonstrate comparable performance to proprietary solutions in structuring pathology report data. This finding has significant implications for healthcare institutions seeking cost-effective, privacy-preserving data structuring solutions. The variations in model performance across different configurations provide valuable insights for practical deployment in pathology departments. Our publicly available bilingual dataset serves as both a benchmark and a resource for future research.Keywords
Funding Information
- Wilhelm Sander-Stiftung (Grand 2022.040.1)
- Bundesministerium für Forschung, Technologie und Raumfahrt (Project FED-PATH)
This publication has 26 references indexed in Scilit:
- A comparative study of large language model-based zero-shot inference and task-specific supervised classification of breast cancer pathology reportsJournal of the American Medical Informatics Association, 2024
- Extracting structured information from unstructured histopathology reports using generative pre‐trained transformer 4 ( GPT ‐4)The Journal of Pathology, 2023
- The future landscape of large language models in medicineCommunications Medicine, 2023
- medBERT.de: A comprehensive German BERT model for the medical domainExpert Systems with Applications, 2023
- Expanded genetic testing of GIST patients identifies high proportion of non-syndromic patients with germline alterationsnpj Precision Oncology, 2023
- Deep learning in cancer pathology: a new generation of clinical biomarkersBritish Journal of Cancer, 2020
- High-accuracy prostate cancer pathology using deep learningNature Machine Intelligence, 2020
- Artificial intelligence in digital pathology — new tools for diagnosis and precision oncologyNature Reviews Clinical Oncology, 2019
- Identification of barriers and facilitators in nationwide implementation of standardized structured reporting in pathology: a mixed method studyVirchows Archiv, 2019
- Effects of text length on lexical diversity measures: Using short texts with less than 200 tokensSystem, 2012