Large Language Models as a Medical Oncology Decision Support Tool: A Retrospective Study Using Real-Life Lung Cancer Cases (Preprint)

Authors:
  • JKJeffrey Karson
    Jeffrey Karson
  • GDGregory Dyson
    Gregory Dyson
  • ASAsmita Shukla
    Asmita Shukla
  • SRSushmitha Nanja Reddy
    Sushmitha Nanja Reddy
  • AZAli Zidan
    Ali Zidan
  • NYNagasai Yalavarthi
    Nagasai Yalavarthi
  • PMPadmapriya Muthu
    Padmapriya Muthu
  • SSSuha Soni
    Suha Soni
  • THTarik Hadid
    Tarik Hadid
  • RMRami Manochakian
    Rami Manochakian
  • ADAnastasios Dimou
    Anastasios Dimou
Preprint
Publisher website
Google Scholar
Abstract
BACKGROUND The integration of artificial intelligence (AI) in clinical oncology has the potential to enhance decision-making by synthesizing vast, evolving datasets. However, the accuracy and clinical utility of large language models (LLMs) for lung cancer treatment recommendations remain unclear. OBJECTIVE This study aimed to evaluate the potential of large language models (GPT-4O and Claude 3 Opus) as clinical decision support tools in medical oncology by assessing their ability to generate accurate, comprehensive, and clinically appropriate treatment recommendations for real-world lung cancer cases. The objectives included measuring the models’ performance across key domains—such as treatment accuracy, comprehensiveness, insight generation, and safety—and determining their suitability for integration into oncology practice. METHODS We conducted a prospective evaluation of GPT-4O (OpenAI) and Claude 3 Opus (Anthropic) in generating treatment recommendations for 25 real-world lung cancer cases from the Fellows Clinic at Karmanos Cancer Institute in Detroit, MI. AI-generated responses were assessed by thoracic oncologists, blinded to the models, using predefined metrics: accuracy, comprehensiveness of oncologic and supportive treatment plans, novel clinical insight, system hallucination or fabrications, and potential for harmful recommendations. Oncologists also reported their confidence in using LLM-generated responses for decision support. RESULTS GPT-4O demonstrated an accuracy score of 4.2 (95% CI, 3.9–4.4), outperforming Claude Opus, which scored 3.6 (95% CI, 3.1–4.1; p = 0.04). GPT-4O was deemed trustworthy in 16 of 25 cases (64%), compared to 11 of 25 cases (44%) for Claude Opus (p = 0.03). Neither model consistently provided novel insights. Both models generated potentially harmful recommendations (GPT-4O: 24%; Claude Opus: 36%) and demonstrated deficiencies in treatment plan comprehensiveness, including omission of maintenance immunotherapy and failure to integrate patient-specific factors such as molecular testing or their specific co-morbidities. CONCLUSIONS While LLMs exhibit moderate accuracy in lung cancer treatment recommendations, critical limitations in comprehensiveness and potential for harmful outputs preclude their standalone use in clinical practice. Domain-specific refinement and rigorous validation are necessary before LLMs can reliably support oncologic decision-making.

This publication has 1 reference indexed in Scilit: