Large Language Models for Automated Structured Information Extraction From CT Reports in Patients With Lymphoma: Retrospective Single-Center Study (Preprint)
Authors:
- Bruno CuriaBCBruno Curia
- Paolo AntonazzoPAPaolo Antonazzo
- Laura BergomiLBLaura Bergomi
- Enea ParimbelliEnea Parimbelli
- Riccardo BellazziRBRiccardo Bellazzi
- Tommaso Mario BuonocoreTBTommaso Mario Buonocore
- Lorenzo AlberghiLALorenzo Alberghi
- Leonardo BrizziLBLeonardo Brizzi
- Chandra BortolottoCBChandra Bortolotto
- Lorenzo PredaLPLorenzo Preda
Preprint
- 29 May 2026
- preprint
- Published by JMIR Publications Inc. in JMIR Preprints
Abstract
BACKGROUND Radiology reports are commonly written as free text, which limits data standardization, automated analysis, and secondary use for research and clinical decision support. Structured reporting templates may improve consistency and interoperability, but their adoption can be limited by increased reporting burden and reduced narrative flexibility. Large language models may support the automatic extraction of structured information from existing free-text radiology reports. OBJECTIVE To compare different large language models (LLMs) for natural language processing of free-text CT reports in patients with lymphoma, assessing their ability to automatically populate a standardized structured reporting (SR) template. METHODS In this single-center retrospective study, 174 Italian CT reports for lymphoma staging were analyzed. The Report section of the SIRM lymphoma CT SR template (65 fields: 19 free-text, 29 numerical, 17 multiple-choice) was used as reference structure. Two radiologists generated reference annotations and inter-annotator agreement was assessed with Krippendorff’s alpha. Claude 3.5 Sonnet, Gemini 1.5 Flash, and Mistral Large 2407 were tested in two settings: full schema and lesion-filtered schema. Performance was evaluated using accuracy, specificity, sensitivity, and F1-score. RESULTS Mean global inter-annotator agreement was 0.75±0.13. All models performed better with lesionfiltered schemas than with full schemas: mean accuracy increased from 0.63–0.64 to 0.72–0.74 and F1-score from 0.52–0.54 to 0.76–0.78, with specificity remaining >0.82. Multiple-choice fields showed the best performance (filtered-schema F1 >0.90; sensitivity ≥0.95), followed by numerical fields (F1 ~0.80), while free-text fields were the most challenging (F1 ~0.57). Claude 3.5 Sonnet achieved the best overall results (accuracy 0.74; F1-score 0.78), closely followed by Gemini 1.5 Flash. CONCLUSIONS General-purpose LLMs can effectively extract structured information from Italian lymphoma CT free-text reports without domain-specific fine-tuning. Schema filtering substantially improves performance and may facilitate integration of LLM-based tools into radiology workflows for SR completion, research data reuse, and clinical decision support.Keywords
This publication has 12 references indexed in Scilit:
- Privacy-Preserving Generation of Structured Lymphoma Progression Reports from Cross-sectional Imaging: A Comparative Analysis of Llama 3.3 and Llama 4Journal of Imaging Informatics in Medicine, 2025
- Extracting structured information from unstructured histopathology reports using generative pre‐trained transformer 4 ( GPT ‐4)The Journal of Pathology, 2023
- ESR paper on structured reporting in radiology—update 2023Insights into Imaging, 2023
- Natural language processing to convert unstructured COVID-19 chest-CT reports into structured reportsEuropean Journal of Radiology Open, 2023
- Natural language processing in radiology: Clinical applications and future directionsClinical Imaging, 2023
- Deep Learning–based Assessment of Oncologic Outcomes from Natural Language Processing of Structured Radiology ReportsRadiology: Artificial Intelligence, 2022
- A systematic review of natural language processing applied to radiology reportsBMC Medical Informatics and Decision Making, 2021
- Structured Reporting: Patient Care Enhancement or Productivity Nightmare?Radiology, 2008
- Inter-Coder Agreement for Computational LinguisticsComputational Linguistics, 2008
- Il referto strutturato2008