Tomohiro Takayama

Large language models (LLMs) have been investigated for clinical documentation, with concerns about hallucinations and factual errors. Clinician review and revision of LLM-generated drafts are considered essential; however, the impact of such a workflow on both documentation time and quality remains unknown. This single-center, parallel-group, prospective, randomized, open-label, blinded-endpoint (PROBE) pilot trial evaluated whether a clinician-in-the-loop workflow improves clinical documentation time and quality compared with clinician-only drafting. From February 18 to March 14, 2025, at the Department of Ophthalmology, Kyoto University Hospital, 21 physicians were randomized, and 17 completed the intervention. Participants were randomized to either the Clinician-in-the-loop group or the Clinician-only group. Participants created discharge summaries and referrals for six simulated patient records. The Clinician-in-the-loop group reviewed and edited LLM-generated drafts, whereas the Clinician-only group drafted documents from scratch using matched templates. Unedited LLM drafts formed the LLM-only group. Seventeen physicians submitted 48 discharge summaries and 48 discharge referrals in the Clinician-in-the-loop group, 54 of each document type in the Clinician-only group, and 48 of each in the LLM-only group. Clinician-in-the-loop showed a shorter creation time for summaries (β = −59.4 seconds; 95% CI, −150.6 to 31.7; P = .18) and a longer creation time for referrals (β = 52.9 seconds; 95% CI, −7.0 to 112.9; P = .08) than Clinician-only, although neither difference was statistically significant. In most expert-rated quality domains, the clinician-in-the-loop workflow outperformed clinician-only drafting. LLM-only drafts were fastest but had the lowest quality. In conclusion, a clinician-in-the-loop approach generally improved document quality and, although the primary analysis showed no statistically significant difference in documentation time, suggested potential time savings for certain document types. These findings highlight the importance of active clinician supervision and support further evaluation of clinician-in-the-loop workflows in real-world clinical settings. Trial Registration ClinicalTrials.gov Identifier NCT07187050 (registered on September 18, 2025).

IMPORTANCE Large language models (LLMs) have been investigated for clinical documentation, with concerns about hallucinations and factual errors. Clinician review and revision of LLM-generated drafts are therefore considered essential, yet the impact of such workflow on both documentation time and quality remains unknown. OBJECTIVE To assess whether physician review and editing of LLM-generated drafts improves the time and quality of clinical documentation compared with clinician-only drafting in a randomized controlled trial setting. DESIGN Single-center, parallel-group, prospective, randomized, open-label, blinded-endpoint (PROBE) pilot trial conducted from February 18 to March 14, 2025. SETTING Kyoto University Hospital, Department of Ophthalmology. PARTICIPANTS Twenty-one ophthalmology physicians were randomized; 17 completed the study, and 4 withdrew before initiating intervention. INTERVENTIONS Participants were randomized to either the Clinician-in-the-loop group or the Clinician-only group. All participants created discharge summaries and referrals for six simulated patient records. In the Clinician-in-the-loop group, drafts were generated with an LLM assistant and then reviewed and edited by participants, whereas in the Clinician-only group, documents were drafted from scratch using matched templates. Unedited LLM drafts were additionally analyzed as the LLM-only group. MAIN OUTCOMES AND MEASURES Document creation time (primary) and expert-rated document quality across six domains plus overall quality (secondary). RESULTS Seventeen physicians submitted 48 discharge summaries and 48 discharge referrals in the Clinician-in-the-loop group, 54 of each document type in the Clinician-only group, and 48 of each in the LLM-only group. For summaries, Clinician-in-the-loop was associated with shorter creation time versus Clinician-only (β = −59.4 seconds; 95% CI, −118.1 to −0.8; P = .047). For referrals, clinician-in-the-loop required more time (β = 94.8 seconds; 95% CI, 40.4 to 149.3; P<.001). In most quality domains for both document types, the clinician-in-the-loop workflow outperformed clinician-only drafting. LLM-only drafts were fastest but had the lowest quality. CONCLUSIONS AND RELEVANCE A clinician-in-the-loop approach improved document quality and accelerated documentation. Active clinician review of LLM-generated drafts is essential for clinical documentation, and such workflows may help enhance working conditions and patient care. TRIAL REGISTRATION ClinicalTrials.gov Identifier NCT07187050 Question Does a workflow in which clinicians review and edit LLM-generated drafts (“clinician-in-the-loop”) improve the efficiency and quality of clinical documentation compared with clinician-only drafting? Findings In this single-center randomized controlled trial including 17 ophthalmology physicians, discharge summaries were significantly faster with the clinician-in-the-loop. Across both document types, clinician-in-the-loop drafts achieved higher quality scores than clinician-only documents. Meaning A clinician-in-the-loop workflow using LLMs can simultaneously enhance the efficiency and quality of clinical documentation; accordingly, active clinician review remains essential for improving working conditions and patient care overall.