How subjective CT image quality assessment becomes surprisingly reliable: pairwise comparisons instead of Likert scale
Open Access
- 2 January 2024
- journal article
- research article
- Published by Springer Nature in European Radiology
- Vol. 34 (7) , 4494-4503
Abstract
Objectives: The aim of this study is to improve the reliability of subjective IQ assessment using a pairwise comparison (PC) method instead of a Likert scale method in abdominal CT scans. Methods: Abdominal CT scans (single-center) were retrospectively selected between September 2019 and February 2020 in a prior study. Sample variance in IQ was obtained by adding artificial noise using dedicated reconstruction software, including reconstructions with filtered backprojection and varying iterative reconstruction strengths. Two datasets (each n = 50) were composed with either higher or lower IQ variation with the 25 original scans being part of both datasets. Using in-house developed software, six observers (five radiologists, one resident) rated both datasets via both the PC method (forcing observers to choose preferred scans out of pairs of scans resulting in a ranking) and a 5-point Likert scale. The PC method was optimized using a sorting algorithm to minimize necessary comparisons. The inter- and intraobserver agreements were assessed for both methods with the intraclass correlation coefficient (ICC). Results: Twenty-five patients (mean age 61 years ± 15.5; 56% men) were evaluated. The ICC for interobserver agreement for the high-variation dataset increased from 0.665 (95%CI 0.396–0.814) to 0.785 (95%CI 0.676–0.867) when the PC method was used instead of a Likert scale. For the low-variation dataset, the ICC increased from 0.276 (95%CI 0.034–0.500) to 0.562 (95%CI 0.337–0.729). Intraobserver agreement increased for four out of six observers. Conclusion: The PC method is more reliable for subjective IQ assessment indicated by improved inter- and intraobserver agreement. Clinical relevance statement: This study shows that the pairwise comparison method is a more reliable method for subjective image quality assessment. Improved reliability is of key importance for optimization studies, validation of automatic image quality assessment algorithms, and training of AI algorithms. Key Points: • Subjective assessment of diagnostic image quality via Likert scale has limited reliability. • A pairwise comparison method improves the inter- and intraobserver agreement. • The pairwise comparison method is more reliable for CT optimization studies. GraphicalKeywords
This publication has 25 references indexed in Scilit:
- Performance of Four Subjective Video Quality Assessment Protocols and Impact of Different Rating Preprocessing and Analysis MethodsIEEE Journal of Selected Topics in Signal Processing, 2016
- Correlation between subjective and objective assessment of magnetic resonance (MR) imagesMagnetic Resonance Imaging, 2016
- Review of medical image quality assessmentBiomedical Signal Processing and Control, 2016
- A Novel Pairwise Comparison-Based Method to Determine Radiation Dose Reduction Potentials of Iterative Reconstruction Algorithms, Exemplified Through Circle of Willis Computed Tomography AngiographyInvestigative Radiology, 2016
- Correlation of clinical and physical-technical image quality in chest CT: a human cadaver study applied on iterative reconstructionBMC Medical Imaging, 2015
- Pairwise Comparison Versus Likert Scale for Biomedical Image AssessmentAmerican Journal of Roentgenology, 2015
- Objective assessment of image quality and dose reduction in CT iterative reconstructionMedical Physics, 2014
- Comparison of Four Subjective Methods for Image Quality AssessmentComputer Graphics Forum, 2012
- Forced choice and ordinal discrete rating assessment of image quality: A comparisonJournal of Digital Imaging, 1997
- A law of comparative judgment.Psychological Review, 1927