Reliability of retrospective medical record review for identifying adverse events: findings from the 2023 Korean Patient Safety Incident Inquiry
- 10 January 2026
- journal article
- research article
- Published by Oxford University Press (OUP) in International Journal for Quality in Health Care
- Vol. 38 (1)
Abstract
Background: Training and engaging various healthcare professionals in the review process is important for improving the feasibility and utility of retrospective medical record reviews in identifying adverse events. This study aimed to evaluate the interrater reliability of retrospective medical record review for identifying adverse events and to examine the feasibility of involving diverse types of healthcare professionals as reviewers. Methods: A three-stage retrospective medical record review was conducted at three medical institutions following the methodology of the Korean Patient Safety Incident Inquiry conducted in South Korea in 2019. In the first stage, the presence or absence of 41 predefined screening criteria—nationally used in South Korea to screen for potential adverse events—was assessed by two nurses or by one nurse and one health information manager in 120 randomly selected patients at each institution. In some cases, external reviewers participated alongside the internal reviewers. In the second stage, if any screening criteria were identified, two physicians or one nurse and physician evaluated the occurrence of adverse events. In the third stage, the occurrence and characteristics of adverse events were finalized. Interrater reliability was assessed for the first and second stages using percentage agreement and kappa statistics. Results: Of 360 reviewed patients across three institutions, 139 (38.6%) were identified by screening criteria, 39 underwent detailed adverse event review, and 47 adverse events were ultimately confirmed by the final committee. The overall agreement rate in the first-stage review was 81.4% (including all negative cases), with a kappa value of 0.558 [95% confidence interval (CI): 0.467–0.649]. In the second-stage review, the agreement rate was 82.0%, with a kappa value of 0.421 (95% CI: 0.223–0.588). The agreement between nurses and health information managers in the first stage was 82.9% (kappa = 0.600, 95% CI: 0.494–0.705). For internal and external reviewers, it was 75.0% (kappa = 0.446, 95% CI: 0.303–0.589). In the second stage, the agreement between physicians and nurses was 82.9% (kappa = 0.600, 95% CI: 0.295–0.848). Conclusion: This study provides evidence supporting the diversification of reviewer professions and types at each stage of retrospective medical record reviews to identify adverse events. However, establishing rigorous procedures to enhance interrater reliability, such as training in medical record review methods and conducting interim assessments during the review process, is essential.Keywords
Funding Information
- Korea Institute for Healthcare Accreditation
This publication has 23 references indexed in Scilit:
- Deep sight: enhancing periprocedural adverse event recording in endoscopy by structuring text documentation with privacy-preserving large language modelsiGIE, 2024
- Variation between hospitals and reviewers in detection of adverse events identified through medical record review in KoreaInternational Journal for Quality in Health Care, 2020
- Impact of Patient Safety Incidents Reported by the General Public in KoreaJournal of Patient Safety, 2020
- A Systematic Review of Methods for Medical Record Analysis to Detect Adverse Events in Hospitalized PatientsJournal of Patient Safety, 2020
- Examining Causes and Prevention Strategies of Adverse Events in Deceased Hospital Patients: A Retrospective Patient Record Review Study in the NetherlandsJournal of Patient Safety, 2019
- Patient and hospital characteristics that influence incidence of adverse events in acute public hospitals in Portugal: a retrospective cohort studyInternational Journal for Quality in Health Care, 2018
- Adverse events in a Tunisian hospital: results of a retrospective cohort studyInternational Journal for Quality in Health Care, 2010
- The incidence of adverse events in Swedish hospitals: a retrospective medical record review studyInternational Journal for Quality in Health Care, 2009
- The Nature of Adverse Events in Hospitalized PatientsNew England Journal of Medicine, 1991
- Incidence of Adverse Events and Negligence in Hospitalized PatientsNew England Journal of Medicine, 1991