Sociotechnical analysis of qualitative data using a large language model-assisted framework with performance compared to a human-only coded reference standard

Authors:
Publisher website
Google Scholar
Abstract
Objective: This methodological validation study evaluated a large language model (LLM), multi-step framework for thematic analysis (TA) of healthcare interviews, compared to a human-only coded reference standard. Materials and Methods: The unit of analysis was deidentified transcripts of interviews with geriatrics patients describing their experiences contributing contextual patient generated health data. An experienced 4-person coding team completed a primarily inductive TA in which 50 codes were synthesized into 3 themes through iterative discussion. LLM-assisted TA was performed using ChatGPT (GPT-4o and GPT-4.5 models). In a 3-step process of code generation, code refinement, and theme generation, 54 initial codes were produced, refined to 25 codes, and consolidated into 3 final themes. Quantitative evaluation of the LLM model performance comparing human-generated and LLM generated codes and themes was performed with sentence-t5-xxl embeddings and a + 0.7 cosine similarity threshold. Results: The cosine-similarity analysis showed a precision of 100% and a recall of 88%. Code-level similarity scores ranged from 0.70 to 0.82 whereas theme-level similarity scores ranged from 0.80 to 0.83, with manual review confirming consistent coding and thematic boundaries. Human-only coded analysis was completed in approximately 39 person-hours compared to 5 person-hours for the LLM analysis. The approximate labor cost for human-only coded analysis was $1924 or $240 per transcript and $237 inclusive of paid subscription ($29.63 per transcript) for the LLM analysis. Discussion: An LLM-assisted multi-step approach can closely replicate human-only TA at reduced time and cost. Conclusion: An LLM-assisted approach offers a practical and scalable tool for qualitative sociotechnical research when combined with human oversight.
Funding Information
  • Agency for Healthcare Research and Quality (1R21HS029982)

This publication has 37 references indexed in Scilit: