Outcome Prediction Using Multi-Modal Information: Integrating Large Language Model-Extracted Clinical Information and Image Analysis
Open Access
Abstract
Survival prediction post-cystectomy is essential for the follow-up care of bladder cancer patients. This study aimed to evaluate artificial intelligence (AI)-large language models (LLMs) for extracting clinical information and improving image analysis, with an initial application involving predicting five-year survival rates of patients after radical cystectomy for bladder cancer. Data were retrospectively collected from medical records and CT urograms (CTUs) of bladder cancer patients between 2001 and 2020. Of 781 patients, 163 underwent chemotherapy, had pre- and post-chemotherapy CTUs, underwent radical cystectomy, and had an available post-surgery five-year survival follow-up. Five AI-LLMs (Dolly-v2, Vicuna-13b, Llama-2.0-13b, GPT-3.5, and GPT-4.0) were used to extract clinical descriptors from each patient’s medical records. As a reference standard, clinical descriptors were also extracted manually. Radiomics and deep learning descriptors were extracted from CTU images. The developed multi-modal predictive model, CRD, was based on the clinical (C), radiomics (R), and deep learning (D) descriptors. The LLM retrieval accuracy was assessed. The performances of the survival predictive models were evaluated using AUC and Kaplan–Meier analysis. For the 163 patients (mean age 64 ± 9 years; M:F 131:32), the LLMs achieved extraction accuracies of 74%~87% (Dolly), 76%~83% (Vicuna), 82%~93% (Llama), 85%~91% (GPT-3.5), and 94%~97% (GPT-4.0). For a test dataset of 64 patients, the CRD model achieved AUCs of 0.89 ± 0.04 (manually extracted information), 0.87 ± 0.05 (Dolly), 0.83 ± 0.06~0.84 ± 0.05 (Vicuna), 0.81 ± 0.06~0.86 ± 0.05 (Llama), 0.85 ± 0.05~0.88 ± 0.05 (GPT-3.5), and 0.87 ± 0.05~0.88 ± 0.05 (GPT-4.0). This study demonstrates the use of LLM model-extracted clinical information, in conjunction with imaging analysis, to improve the prediction of clinical outcomes, with bladder cancer as an initial example.Keywords
Funding Information
- National Institutes of Health (U01-CA232931)
This publication has 35 references indexed in Scilit:
- Integrative radiomics and transcriptomics analyses reveal subtype characterization of non-small cell lung cancerEuropean Radiology, 2023
- Artificial intelligence: A promising frontier in bladder cancer diagnosis and outcome predictionCritical Reviews in Oncology/Hematology, 2022
- Impact of Molecular Subtypes in Muscle-invasive Bladder Cancer on Predicting Response and Survival after Neoadjuvant ChemotherapyEuropean Urology, 2017
- Feature selection using Joint Mutual Information MaximisationExpert Systems with Applications, 2015
- Prediction of mortality after radical cystectomy for bladder cancer by machine learning techniquesComputers in Biology and Medicine, 2015
- Prognostic and Prediction Tools in Bladder Cancer: A Comprehensive Review of the LiteratureEuropean Urology, 2015
- Nomogram for predicting survival in patients with unresectable and/or metastatic urothelial cancer who are treated with cisplatin‐based chemotherapyCancer, 2013
- Auto-Initialized Cascaded Level Set (AI-CALS) Segmentation of Bladder Lesions on Multidetector Row CT UrographyAcademic Radiology, 2013
- Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed dataStatistics in Medicine, 1998
- Back-propagation neural networks for modeling complex systemsArtificial Intelligence in Engineering, 1995