An external peer review instrument for Korean Medicine clinical practice guideline drafts (KM-EPRI): development and content validation

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Background: Korean Medicine clinical practice guidelines (KM-CPGs) undergo mandatory external peer review of their drafts during development, yet reviewers lack a standardized, validated instrument to guide this task. Widely used appraisal tools assess the quality of completed guidelines rather than support the section-level review of a draft. We aimed to develop an external peer review instrument for KM-CPG drafts (KM-EPRI) and to assess its content validity. Methods: In this two-phase methodological study, a draft instrument was first derived by directed qualitative content analysis of the external review comments produced for previously published KM-CPGs, with each comment coded into one of four predefined review modules: the clinical questions (CQ), the introductory components (CPG-I), the development methods (CPG-M), and the recommendations (CPG-R). The content validity and acceptability of the items were then assessed through a modified Delphi study with a multidisciplinary panel of KM clinicians and evidence-based medicine methodologists. Consensus was defined a priori as an item-level content validity index (I-CVI) ≥ 0.80. The scale-level content validity index (S-CVI/Ave), Kendall’s coefficient of concordance ( W ), and the interquartile range (IQR) were reported as supporting measures. Results: From 3,630 external review comments, 2,073 content-related comments were consolidated and refined into a finalized instrument of 117 review items across 4 modules and 33 subdomains. All 10 invited experts participated (response rate 100%, no attrition), and every item reached the consensus threshold in a single round under a pre-defined early-stopping rule. All 117 items had an I-CVI ≥ 0.80, the S-CVI/Ave was 0.976, and the IQR was ≤ 1 for every item (median 0.75). Kendall’s W was low (0.196), reflecting a ceiling effect from uniformly high ratings rather than disagreement among panelists. Conclusion: KM-EPRI can serve as a consensus-based, structured reference for the external peer review of KM-CPG drafts during development. Its effect on review quality and inter-rater reliability remains to be evaluated in practice.

This publication has 19 references indexed in Scilit: