Rachel Dalrymple

Objective: This methodological validation study evaluated a large language model (LLM), multi-step framework for thematic analysis (TA) of healthcare interviews, compared to a human-only coded reference standard. Materials and Methods: The unit of analysis was deidentified transcripts of interviews with geriatrics patients describing their experiences contributing contextual patient generated health data. An experienced 4-person coding team completed a primarily inductive TA in which 50 codes were synthesized into 3 themes through iterative discussion. LLM-assisted TA was performed using ChatGPT (GPT-4o and GPT-4.5 models). In a 3-step process of code generation, code refinement, and theme generation, 54 initial codes were produced, refined to 25 codes, and consolidated into 3 final themes. Quantitative evaluation of the LLM model performance comparing human-generated and LLM generated codes and themes was performed with sentence-t5-xxl embeddings and a + 0.7 cosine similarity threshold. Results: The cosine-similarity analysis showed a precision of 100% and a recall of 88%. Code-level similarity scores ranged from 0.70 to 0.82 whereas theme-level similarity scores ranged from 0.80 to 0.83, with manual review confirming consistent coding and thematic boundaries. Human-only coded analysis was completed in approximately 39 person-hours compared to 5 person-hours for the LLM analysis. The approximate labor cost for human-only coded analysis was $1924 or $240 per transcript and $237 inclusive of paid subscription ($29.63 per transcript) for the LLM analysis. Discussion: An LLM-assisted multi-step approach can closely replicate human-only TA at reduced time and cost. Conclusion: An LLM-assisted approach offers a practical and scalable tool for qualitative sociotechnical research when combined with human oversight.

Background: Lung cancer remains the leading cause of cancer-related mortality worldwide, with low-dose computed tomography screening demonstrating an approximately 20% reduction in mortality among high-risk individuals. Despite this benefit, screening prevalence remains suboptimal, with often less than 20% of eligible individuals reported to be up to date on screening. Shared decision-making is essential for effective lung cancer screening (LCS) implementation, with decision aids shown to enhance patient knowledge and engagement. Objective: The aim of this study is to identify patient preferences, concerns, and design considerations through qualitative evaluation of MyLungHealth, a personalized patient-facing educational tool for LCS integrated with electronic health records, and to describe how these findings informed iterative design modifications. Methods: We employed qualitative research methods through focus groups (n=34) and individual interviews (n=18) with individuals who met screening eligibility criteria. Participants were recruited from the University of Utah Health and New York University Langone Health between May and December 2023. Feedback was analyzed using Braun and Clarke’s thematic analysis principles. Results: Six themes were organized into three overarching domains. Domain A included interpretation and impact of personalized risk information: theme 1, difficulties interpreting risk information, and theme 2, varied impacts of risk information on motivation. Domain B included autonomy, privacy, and user interface preferences: theme 3, desire for autonomy and control over personal health data, and theme 4, preference for straightforward language and multiple information formats. Domain C included integration with clinical workflows and patient portal systems: theme 5, expectations for integration with health care provider workflows, and theme 6, mixed experiences with personal health record systems. These insights led to key design modifications, including simplified risk presentation, multimodal content delivery options (video and text), and implementation of electronic health record alerts for clinicians. Conclusions: The user-centered design process for MyLungHealth revealed important considerations for developing effective patient education tools for LCS. The findings highlighted the need for simplified risk presentation, personalized information delivery, and integration with clinical workflows. These findings underscore the importance of balancing comprehensive risk communication with user accessibility.

Introduction: The 2-1-1 system is a nationwide social service provider that connects callers with unmet needs to appropriate community resources. Every year, over two million Americans experiencing food insecurity (FI) seek help from 2-1-1. Yet, little is known about the FI experiences among 2-1-1 callers seeking food-related support, specifically, who they are, the circumstances that lead them to contact 2-1-1, and the strategies they believe could help reduce FI. Methods: To address this gap, we interviewed 30 food-related callers (20 in English and 10 in Spanish) to the Utah 2-1-1 system between November 2024 and January 2025. Results: Thematic analysis revealed three primary reasons for seeking food-related help from 2-1-1: (1) ineligibility for federal nutrition and financial assistance programs; (2) facing non-income-related barriers to accessing food; and (3) experiencing unexpected life events. Callers suggested strengthening and adapting federal nutrition assistance programs, improving emergency food provision systems, and enhancing informational support. Conclusion: Based on these insights, we provide suggestions on how to leverage the 2-1-1 system to intervene on FI.