Evolution of publicly available large language models for complex decision-making in breast cancer care
Open Access
- 29 May 2024
- journal article
- research article
- Published by Springer Nature in Archives of Gynecology and Obstetrics
- Vol. 310 (1) , 537-550
Abstract
Purpose: This study investigated the concordance of five different publicly available Large Language Models (LLM) with the recommendations of a multidisciplinary tumor board regarding treatment recommendations for complex breast cancer patient profiles. Methods: Five LLM, including three versions of ChatGPT (version 4 and 3.5, with data access until September 3021 and January 2022), Llama2, and Bard were prompted to produce treatment recommendations for 20 complex breast cancer patient profiles. LLM recommendations were compared to the recommendations of a multidisciplinary tumor board (gold standard), including surgical, endocrine and systemic treatment, radiotherapy, and genetic testing therapy options. Results: GPT4 demonstrated the highest concordance (70.6%) for invasive breast cancer patient profiles, followed by GPT3.5 September 2021 (58.8%), GPT3.5 January 2022 (41.2%), Llama2 (35.3%) and Bard (23.5%). Including precancerous lesions of ductal carcinoma in situ, the identical ranking was reached with lower overall concordance for each LLM (GPT4 60.0%, GPT3.5 September 2021 50.0%, GPT3.5 January 2022 35.0%, Llama2 30.0%, Bard 20.0%). GPT4 achieved full concordance (100%) for radiotherapy. Lowest alignment was reached in recommending genetic testing, demonstrating a varying concordance (55.0% for GPT3.5 January 2022, Llama2 and Bard up to 85.0% for GPT4). Conclusion: This early feasibility study is the first to compare different LLM in breast cancer care with regard to changes in accuracy over time, i.e., with access to more data or through technological upgrades. Methodological advancement, i.e., the optimization of prompting techniques, and technological development, i.e., enabling data input control and secure data processing, are necessary in the preparation of large-scale and multicenter studies to provide evidence on their safe and reliable clinical application. At present, safe and evidenced use of LLM in clinical breast cancer care is not yet feasible.Keywords
Funding Information
- Philipps-Universität Marburg
This publication has 35 references indexed in Scilit:
- Evaluating ChatGPT as an adjunct for the multidisciplinary tumor board decision-making in primary breast cancer casesArchives of Gynecology and Obstetrics, 2023
- Large language model (ChatGPT) as a support tool for breast tumor boardnpj Breast Cancer, 2023
- Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewersnpj Digital Medicine, 2023
- Antibody-Drug Conjugates in Breast Cancer: Searching for Magic BulletsJournal of Clinical Oncology, 2023
- Self-collection of capillary blood and saliva to determine COVID-19 vaccine immunogenicity in patients with immune-mediated inflammatory diseases and health professionalsFrontiers in Public Health, 2022
- Combined Focused Next-Generation Sequencing Assays to Guide Precision Oncology in Solid Tumors: A Retrospective Analysis from an Institutional Molecular Tumor BoardCancers, 2022
- Translating “Big Data” in Oncology for Clinical Benefit: Progress or ParalysisCancer Research, 2022
- Endocrine Therapy for Ductal Carcinoma In Situ (DCIS) of the Breast with Breast Conserving Surgery (BCS) and Radiotherapy (RT): a Meta-AnalysisPathology and Oncology Research, 2018
- Beyond Dr. Google: the evidence on consumer-facing digital tools for diagnosisDiagnosis, 2018
- Variation in treatment and survival of older patients with non-metastatic breast cancer in five European countries: a population-based cohort study from the EURECCA Breast Cancer GroupBritish Journal of Cancer, 2018