It is Too Many Options: Pitfalls of Multiple-Choice Questions in Generative AI and Medical Education

Authors:
  • SSShrutika Singh
    Shrutika Singh
  • AAAnton Alyakin
    Anton Alyakin
  • DADaniel Alexander Alber
    Daniel Alexander Alber
  • JSJaden Stryker
    Jaden Stryker
  • ATAi Phuong S Tong
    Ai Phuong S Tong
  • KSKarl Sangwon
    Karl Sangwon
  • NGNicolas Goff
    Nicolas Goff
  • MPMathew de la Paz
    Mathew de la Paz
  • MHMiguel Hernandez-Rovira
    Miguel Hernandez-Rovira
  • KPKi Yun Park
    Ki Yun Park
  • ELEric Claude Leuthardt
    Eric Claude Leuthardt
  • EOEric Karl Oermann
    Eric Karl Oermann
Preprint
Publisher website
Google Scholar
Abstract
The performance of Large Language Models (LLMs) on multiple-choice question (MCQ) benchmarks is frequently cited as proof of their medical capabilities. We hypothesized that LLM performance on medical MCQs may in part be illusory and driven by factors beyond medical content knowledge and reasoning capabilities. To assess this, we created a novel benchmark of free-response questions with paired MCQs (FreeMedQA). Using this benchmark, we evaluated three state-of-the-art LLMs (GPT-4o, GPT-3.5, and LLama-3-70B-instruct) and found an average absolute deterioration of 39.43% in performance on free-response questions relative to multiple-choice (p = 1.3 * 10-5) which was greater than the human performance decline of 22.29%. To isolate the role of the MCQ format on performance, we performed a masking study, iteratively masking out parts of the question stem. At 100% masking, the average LLM multiple-choice performance was 6.70% greater than random chance (p = 0.002) with one LLM (GPT-4o) obtaining an accuracy of 37.34%. Notably, for all LLMs the free-response performance was near zero. Our results highlight the shortcomings in medical MCQ benchmarks for overestimating the capabilities of LLMs in medicine, and, broadly, the potential for improving both human and machine assessments using LLM-evaluated free-response questions.
All Related Versions