Statistical testing of agreement in overlap-based performance between an AI segmentation device and a multi-expert human panel without requiring a reference standard
- 1 September 2025
- journal article
- Published by SPIE-Intl Soc Optical Eng in Journal of Medical Imaging
- Vol. 12 (05) , 1-20
Abstract
PurposeArtificial intelligence (AI)-based medical imaging devices often include lesion or organ segmentation capabilities. Existing methods for segmentation performance evaluation compare AI results with an aggregated reference standard using accuracy metrics such as the Dice coefficient or Hausdorff distance. However, these approaches are limited by lacking a gold standard and challenges in defining meaningful success criteria. To address this, we developed a statistical method to assess agreement between an AI device and multiple human experts without requiring a reference standard.ApproachWe propose a paired-testing method to evaluate whether an AI device’s segmentation performance significantly differs from that of multiple human experts. The method compares device-to-expert dissimilarity with expert-to-expert dissimilarity, avoiding the need for a reference standard. We validated the method through (1) statistical simulations where the Dice coefficient performance is either shared (“overlap agreeable”) or not shared (“overlap disagreeable”) between the device and experts; (2) image-based simulations using 2D contours with shared or nonshared transformation parameters (transformation agreeable or disagreeable). We also applied the method to compare an AI segmentation algorithm with four radiologists using data from the Lung Image Database Consortium.ResultsStatistical simulations show the method controls type I error (∼0.05) for overlap-agreeable and type II error (∼0) for overlap-disagreeable scenarios. Image-based simulations show acceptable performance with a mean type I error of 0.07 (SD 0.03) for transformation-agreeable and a mean type II error of 0.07 (SD 0.18) for transformation-disagreeable cases.ConclusionsThe paired-testing method offers a new tool for assessing the agreement between an AI segmentation device and multiple human expert panelists without requiring a reference standard.Keywords
This publication has 14 references indexed in Scilit:
- Testing for Interchangeability of Imaging TestsAcademic Radiology, 2014
- The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): A Completed Reference Database of Lung Nodules on CT ScansMedical Physics, 2011
- A multidimensional segmentation evaluation for medical image dataComputer Methods and Programs in Biomedicine, 2009
- Assessing Individual AgreementJournal of Biopharmaceutical Statistics, 2007
- A Permutation Test for Comparing ROC Curves in Multireader Studies: A Multi-reader ROC, Permutation TestAcademic Radiology, 2006
- A framework for evaluating image segmentation algorithmsComputerized Medical Imaging and Graphics, 2006
- Assessing the Agreement Between Two Quantitative Assays with Repeated MeasurementsJournal of Biopharmaceutical Statistics, 2004
- Statistical validation of image segmentation quality based on a spatial overlap index1Academic Radiology, 2004
- Can electronic medical images replace hard-copy film? Defining and testing the equivalence of diagnostic testsStatistics in Medicine, 2001
- STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENTThe Lancet, 1986