WMWalter Stan Mathis

Walter Stan Mathis

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Walter Stan Mathis has a total of 88 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Evaluating Diagnostic Accuracy and Clinical Reasoning of Multiple Large Language Models in Psychiatry

    by Kevin W. Jin, Yasna Rostam-Abadi, Pooja Chaudhary, Margaret A. Garrett, Ashley S. Huang, Mario Montelongo, Caesa Nagpal, Jasperina Shei, Judah Weathers, Juliana S. Zhang, et al.

    Summary: Background: Existing large language model (LLM) evaluations rely on accuracy benchmarks that fail to capture whether models reason well while making diagnoses. Studies that do analyse reasoning focus on post hoc explanations accompanying model outputs rather than distinct, clinician-visible artifacts such as detailed reasoning traces. This creates a translational gap in domains such as psychiatry, where diagnosis relies on narrative interpretation, diagnostic reasoning, and clinical judgment under uncertainty. Methods: We conducted a mixed-methods evaluation of four state-of-the-art LLMs using a clinician-curated dataset of 196 psychiatric case vignettes, including 135 published cases and 61 novel clinician-authored vignettes. Diagnostic accuracy was assessed using multiple metrics (top-1 accuracy, top-5 accuracy, recall@5, and mean reciprocal rank) based on ranked lists of five differential diagnoses per vignette. Clinical reasoning quality was evaluated on a randomly selected subset of 30 vignettes through clinician assessment of model-generated diagnostic reasoning traces alongside qualitative commentary from board-certified psychiatrists. We examined the association between clinician-rated reasoning quality and diagnostic correctness and included an illustrative comparison with psychiatry residents. Findings: Clinician-rated diagnostic reasoning quality was strongly associated with diagnostic correctness in mixed-effects logistic regression analyses (β = 1·80; p < 0·001), whereas data extraction quality alone was not. Across the full vignette set, models demonstrated moderate to high diagnostic accuracy. The highest-performing model achieved a top-5 accuracy of 0·801 and also received the highest clinician-rated reasoning scores. In an illustrative comparison, model diagnostic accuracy fell within the range observed for psychiatry residents. Interpretation: Diagnostic reasoning quality captures clinically meaningful variation in LLM performance beyond accuracy metrics. Psychiatry may represent a stringent testbed for evaluating reasoning in narrative-driven clinical domains. Evaluation frameworks for LLM-based clinical decision support should incorporate structured assessment of reasoning processes, not accuracy alone. Research in Context: Evidence before this study: We searched PubMed and Scopus for studies evaluating large language models for psychiatric diagnosis and/or differential diagnosis from text vignettes. Searches were run from database inception to February 6, 2026, using terms including (“large language model” OR LLM “artificial intelligence” OR “generative AI” OR AI OR ChatGPT OR GPT OR Claude OR Gemini OR DeepSeek OR Llama) AND (psychiatr* OR mental OR DSM OR “differential diagnosis” OR diagnos*) AND (vignette OR case OR “case report”). We included empirical studies that evaluated model diagnostic outputs using psychiatric cases/vignettes and excluded editorials/commentaries and studies that did not report case-level diagnostic performance. We did not formally assess risk of bias/quality; studies were heterogeneous in vignette sources, model access, and outcome definitions, precluding quantitative pooling. Prior studies have typically examined small vignette sets, focused on narrow diagnostic domains, evaluated single models, or relied primarily on outcome-based accuracy metrics. When diagnostic reasoning has been assessed, it has usually been inferred from post hoc explanations accompanying model outputs rather than evaluated as a distinct, clinician-visible artifact. Clinician-grounded evaluations of diagnostic reasoning across multiple contemporary models remain limited. Added value of this study: This study provides a large-scale, clinician-grounded evaluation of diagnostic accuracy and diagnostic reasoning quality across four contemporary large language models using a diverse dataset of psychiatric case vignettes. Rather than relying solely on outcome-based explanations, we directly evaluated model-generated diagnostic reasoning traces as clinician-visible artifacts using structured clinician ratings and qualitative analysis. By integrating multiple accuracy metrics with clinician assessment of reasoning coherence, flexibility, and plausibility, we demonstrate that clinician-rated reasoning quality is strongly associated with diagnostic correctness, whereas data extraction quality alone is not. Our analysis also identifies recurrent reasoning failure modes not captured by accuracy metrics, highlighting psychiatry as a stringent testbed for evaluating reasoning in narrative-driven clinical domains. Implications of all the available evidence: Evaluations of large language models for clinical decision support should extend beyond accuracy to include systematic assessment of clinician-visible diagnostic reasoning. Mixed-methods, clinician-grounded evaluation frameworks that examine both diagnostic outcomes and reasoning artifacts may be critical for responsible assessment of LLMs in psychiatry and other areas of medicine where diagnosis depends on interpretation, judgment, and tolerance of uncertainty.


  • Prescriber Specialty Involvement in Medicare Patients With Chronic Hepatitis B

    by Xiaohan Ying, Nicole Ng, Deirdre Reidy, Jade Azari, Russell Rosenblatt, Walter S. Mathis, Stephen Congly, Arun Jesudian

    Chronic hepatitis B (CHB) is a major cause of liver‐related morbidity and mortality in the United States. Patients with CHB require long‐term antiviral treatment and consistent follow‐up, but often face numerous barriers to accessing care and medications. In this study, we used the Medicare Part D database and the Rural–Urban Continuum code to explore specialty and geographic characteristics of healthcare providers that manage Medicare patients with CHB. Between 2013 and 2021, more than 7000 prescribers prescribed over 2.4 million 30‐day prescriptions of these CHB therapies, of which 2 million (85%) were in metropolitan counties. The number of 30‐day prescriptions increased by 5.4% annually. The number of prescriptions by GI increased by 12.5% a year and prescriptions by APPs increased by 12.2% a year, while prescriptions by ID decreased by 14.0% annually. In non‐metropolitan counties, APPs experienced −5% APC between 2013 and 2021, and PCPs experienced −12.5% APC between 2016 and 2021. In this study, we found that there has been a noticeable shift in the specialties prescribing medications for patients with hepatitis B. Gastroenterologists and APPs became significantly more involved as prescribers. The increase in APP management of patients with CHB is a welcoming development, especially in light of the physician workforce shortage. It is important to create solutions such as co‐management to ensure that patients with CHB receive consistent care without further contributing to supply–demand mismatch in gastroenterology.

  • Open Access

    Geographic access to community pharmacies based on walking, driving, and public transportation in the 10 most populated U.S. areas

    by Nasser Sharareh, Shangbin Tang, Adam Bress, Walter S. Mathis, Lucas A. Berenbrok, Inmaculada Hernandez


  • Open Access

    Codebook Reduction and Saturation: Novel observations on Inductive Thematic Saturation for Large Language Models and initial coding in Thematic Analysis

    by Stefano De Paoli, Walter Stan Mathis

    This paper reflects on the process of performing Thematic Analysis with Large Language Models (LLMs). Specifically, the paper deals with the problem of analytical saturation of initial codes, as produced by LLMs. Thematic Analysis is a well-established qualitative analysis method composed of interlinked phases. A key phase is the initial coding, where the analysts assign labels to discrete components of a dataset. Saturation is a way to measure the validity of a qualitative analysis and relates to the recurrence and repetition of initial codes. In the paper we reflect on how well LLMs achieve analytical saturation and propose also a novel technique to measure Inductive Thematic Saturation (ITS). This novel technique leverages a programming framework called DSPy. The proposed novel approach allows a precise measurement of ITS.

  • Top Cited Papers
    Open Access

    Vulnerability Index Approach to Identify Pharmacy Deserts and Keystone Pharmacies

    by Walter S. Mathis, Lucas A. Berenbrok, Peter A. Kahn, Giovanni Appolon, Shangbin Tang, Inmaculada Hernandez

    Importance: Community pharmacies are crucial for public health, providing essential services such as medication dispensing, vaccinations, and point-of-care testing. Addressing disparities in pharmacy access, particularly in underserved rural and low-income areas, is critical for health equity.Objective: To identify areas in the US at risk of becoming pharmacy deserts through the development of a novel pharmacy vulnerability index.Design, Setting, and Participants: This population-based cross-sectional study in the contiguous 48 states performed geographic information systems analysis of pharmacy data from the National Council for Prescription Drug Programs (NCPDP) dataQ. Participants included all open-door pharmacies (community or retail pharmacies open to the general public without restrictions on who can access its services) in the US as of February 2024. Statistical analysis was performed from July to August 2024.Exposure: The primary exposure was travel time to pharmacies across the US.Main Outcomes and Measures: A pharmacy desert was defined as a census tract where the travel time to the nearest pharmacy exceeds the supermarket access time for that region and urbanicity level. Building on this definition, a pharmacy vulnerability index was developed, which indicates the number of pharmacies that would need to close for a census tract to become a pharmacy desert. Tracts with a pharmacy vulnerability index of 1, depending solely on a single pharmacy for access, were identified as at risk of becoming deserts. Subpopulation totals and percentages living in pharmacy deserts or relying on keystone pharmacies were computed, and then stratified by urbanicity and race.Results: Among 321.3 million individuals (39.7 million [12.3%] Black, 59.0 million [18.2%] Hispanic, 195.0 million [60.3%] White) in the contiguous US, 57.1 million (17.7%) were identified as living in pharmacy deserts, with 28.9 million (8.9%) additionally relying on a single pharmacy for access. Small rural areas were particularly affected, with a higher dependency on single pharmacies (4.1 million individuals [14.3%]).Conclusions and Relevance: In this cross-sectional study of pharmacy access in the US, significant disparities in pharmacy access were identified, especially pronounced in small rural areas. Targeted policy interventions, such as incremental reimbursement rates or other monetary incentives, are needed to ensure the financial sustainability of pharmacies that serve as the sole source of pharmacy services in at-risk areas.


  • Open Access

    Reflections on inductive thematic saturation as a potential metric for measuring the validity of an inductive thematic analysis with LLMs

    by Stefano De Paoli, Walter S. Mathis

    This paper presents a set of reflections on saturation and the use of Large Language Models (LLMs) for performing Thematic Analysis (TA). The paper suggests that initial thematic saturation (ITS) could be used as a metric to assess part of the transactional validity of TA with LLM, focusing on the initial coding. The paper presents the initial coding of two datasets of different sizes, and it reflects on how the LLM reaches some form of analytical saturation during the coding. The procedure proposed in this work leads to the creation of two codebooks, one comprising the total cumulative initial codes and the other the total unique codes. The paper proposes a metric to synthetically measure ITS using a simple mathematical calculation employing the ratio between slopes of the unique and total codes. The paper contributes to the initial body of work exploring how to perform qualitative analysis with LLMs.


  • Open Access

    TimelinePTC: Development of a unified interface for pathways to care collection, visualization, and collaboration in first episode psychosis

    by Walter S Mathis, Maria Ferrara, John Cahill, Sneha Karmani, Sümeyra N Tayfur, Vinod Srihari

    This paper presents TimelinePTC, a web-based tool developed to improve the collection and analysis of Pathways to Care (PTC) data in first episode psychosis (FEP) research. Accurately measuring the duration of untreated psychosis (DUP) is essential for effective FEP treatment, requiring detailed understanding of the patient’s journey to care. However, traditional PTC data collection methods, mainly manual and paper-based, are time-consuming and often fail to capture the full complexity of care pathways. TimelinePTC addresses these limitations by providing a digital platform for collaborative, real-time data entry and visualization, thereby enhancing data accuracy and collection efficiency. Initially created for the Specialized Treatment Early in Psychosis (STEP) program in New Haven, Connecticut, its design allows for straightforward adaptation to other healthcare contexts, facilitated by its open-source codebase. The tool significantly simplifies the data collection process, making it more efficient and user-friendly. It automates the conversion of collected data into a format ready for analysis, reducing manual transcription errors and saving time. By enabling more detailed and consistent data collection, TimelinePTC has the potential to improve healthcare access research, supporting the development of targeted interventions to reduce DUP and improve patient outcomes.


  • Racial and ethnic inequities in spatial access to pharmacies: A geographic information system analysis

    by Giovanni Appolon, Shangbin Tang, Nico Gabriel, Jasmine Morales, Lucas A. Berenbrok, Andrea Z. LaCroix, Jingchuan Guo, Walter S. Mathis, Inmaculada Hernandez