Leveraging large language models to derive multiple sclerosis progression assessments from clinical notes: A feasibility study

Authors:
  • STSunil Thomas
    Sunil Thomas
  • HWHeather Williams
    Heather Williams
  • THTom Hutchinson
    Tom Hutchinson
  • ESEmily Schriver
    Emily Schriver
  • ABAshley Batugo
    Ashley Batugo
  • ABAmit Bar-Or
    Amit Bar-Or
  • VSVishakha Sharma
    Vishakha Sharma
  • FBFrederik Buijs
    Frederik Buijs
  • CPChristopher Perrone
    Christopher Perrone
  • DMDanielle Mowery
    Danielle Mowery
Preprint
Publisher website
Google Scholar
Abstract
This feasibility study assesses using Large Language Models (LLMs) to automate the extraction of Multiple Sclerosis (MS) progression metrics from unstructured clinical notes. While the Expanded Disability Status Scale (EDSS) is the standard for measuring disability, calculating it during routine care is burdensome, often leaving progression details locked in narrative text. Using OpenAI’s GPT-4.1 in a zero-shot setting, the study processed 288 progress notes from 239 patients. The model extracted scores for eight Functional Systems (FS) and derived a total EDSS score. Performance was evaluated against expert annotations using a strict and lenient agreement criteria. Results showed strong performance for FS extraction, with accuracy ranging from 0.782 to 0.971 across domains. While exact EDSS agreement was variable due to documentation ambiguity, applying clinically meaningful tolerance bands yielded high consistency, with most disagreements being minor adjacent errors unlikely to alter therapy. The study confirms that LLMs can feasibly derive detailed disability measures, potentially reducing documentation burden enabling the automated, population-wide tracking of MS progression over time.