Development and Establishment of Initial Validity Evidence for a Novel Tool for Assessing Trainee Admission Notes
- 28 January 2020
- journal article
- research article
- Published by Springer Nature in Journal of General Internal Medicine
- Vol. 35 (4) , 1078-1083
Abstract
Background Documentation is a key component of practice, yet few curricula have been published to teach trainees proper note construction. Additionally, a gold standard for assessing note quality does not exist, and no documentation assessment tools integrate with established competency-based frameworks. Objective To develop and establish initial validity evidence for a novel tool that assesses key components of trainee admission notes and maps to the Accreditation Council for Graduate Medical Education (ACGME) milestone framework. Design Using an iterative, consensus building process we developed the Admission Note Assessment Tool (ANAT). Pilot testing was performed with both the supervising attending and study team raters not involved in care of the patients. The finalized tool was piloted with attendings from other institutions. Participants Local experts participated in tool development and pilot testing. Additional attending physicians participated in pilot testing. Main Measures Content, response process, and internal structure validity evidence was gathered using Messick’s framework. Inter-rater reliability was assessed using percent agreement. Key Results The final tool consists of 16 checklist items and two global assessment items. Pilot testing demonstrated rater agreement of 72% to 100% for checklist items and 63% to 70% for global assessment items. Note assessment required an average of 12.3 min (SD 3.7). The study generated validity evidence in the domains of content, response process, and internal structure for use of the tool in rating admission notes. Conclusions The ANAT assesses individual components of a note, incorporates billing criteria, targets note “bloat,” allows for narrative feedback, and provides global assessments mapped to the ACGME milestone framework. The ANAT can be used to assess admission notes by any attending and at any time after note completion with minimal rater training. The ANAT allows programs to implement routine note assessment for multiple functions with the use of a single tool.Keywords
This publication has 21 references indexed in Scilit:
- Toward Defining the Foundation of the MD Degree: Core Entrustable Professional Activities for Entering ResidencyAcademic Medicine, 2016
- Resident Notes in an Electronic Health RecordClinical Pediatrics, 2016
- Clinical Documentation in the 21st Century: Executive Summary of a Policy Position Paper From the American College of PhysiciansAnnals of Internal Medicine, 2015
- The effectiveness of a bundled intervention to improve resident progress notes in an electronic health recordJournal of Hospital Medicine, 2014
- Electronic health records improve clinical note qualityJournal of the American Medical Informatics Association, 2014
- QNOTE: an instrument for measuring the quality of EHR clinical notesJournal of the American Medical Informatics Association, 2014
- Assessing Electronic Note Quality Using the Physician Documentation Quality Instrument (PDQI-9)Applied Clinical Informatics, 2012
- Physicians’ Attitudes Towards Copy and Pasting in Electronic Note WritingJournal of General Internal Medicine, 2008
- Preliminary Development of the Physician Documentation Quality InstrumentJournal of the American Medical Informatics Association, 2008
- Impacts of Computerized Physician Documentation in a Teaching Hospital: Perceptions of Faculty and Resident PhysiciansJournal of the American Medical Informatics Association, 2004