Assessing clinical efficacy of polyp detection models using open-access datasets
Authors:
Open Access
- 1 August 2024
- journal article
- research article
- Published by Frontiers Media SA in Frontiers in Oncology
- Vol. 14, 1422942
Abstract
Background: Ensuring accurate polyp detection during colonoscopy is essential for preventing colorectal cancer (CRC). Recent advances in deep learning-based computer-aided detection (CADe) systems have shown promise in enhancing endoscopists’ performances. Effective CADe systems must achieve high polyp detection rates from the initial seconds of polyp appearance while maintaining low false positive (FP) detection rates throughout the procedure.Method: We integrated four open-access datasets into a unified platform containing over 340,000 images from various centers, including 380 annotated polyps, with distinct data splits for comprehensive model development and benchmarking. The REAL-Colon dataset, comprising 60 full-procedure colonoscopy videos from six centers, is used as the fifth dataset of the platform to simulate clinical conditions for model evaluation on unseen center data. Performance assessment includes traditional object detection metrics and new metrics that better meet clinical needs. Specifically, by defining detection events as sequences of consecutive detections, we compute per-polyp recall at early detection stages and average per-patient FPs, enabling the generation of Free-Response Receiver Operating Characteristic (FROC) curves.Results: Using YOLOv7, we trained and tested several models across the proposed data splits, showcasing the robustness of our open-access platform for CADe system development and benchmarking. The introduction of new metrics allows for the optimization of CADe operational parameters based on clinically relevant criteria, such as per-patient FPs and early polyp detection. Our findings also reveal that omitting full-procedure videos leads to non-realistic assessments and that detecting small polyp bounding boxes poses the greatest challenge.Conclusion: This study demonstrates how newly available open-access data supports ongoing research progress in environments that closely mimic clinical settings. The introduced metrics and FROC curves illustrate CADe clinical efficacy and can aid in tuning CADe hyperparameters.Keywords
This publication has 27 references indexed in Scilit:
- Sources of performance variability in deep learning-based polyp detectionInternational Journal of Computer Assisted Radiology and Surgery, 2023
- Performance and attitudes toward real-time computer-aided polyp detection during colonoscopy in a large tertiary referral center in the United StatesGastrointestinal Endoscopy, 2023
- Framework and metrics for the clinical use and implementation of artificial intelligence algorithms into endoscopy practice: recommendations from the American Society for Gastrointestinal Endoscopy Artificial Intelligence Task ForceGastrointestinal Endoscopy, 2023
- A multi-centre polyp detection and segmentation dataset for generalisability assessmentScientific Data, 2023
- Hyper-parameter optimization of deep learning architectures using artificial bee colony (ABC) algorithm for high performance real-time automatic colorectal cancer (CRC) polyp detectionApplied Intelligence, 2022
- Gorilla in the room: Even experts can miss polyps at colonoscopy and how AI helps complex visual perception tasksDigestive and Liver Disease, 2022
- Computer-aided detection versus advanced imaging for detection of colorectal neoplasia: a systematic review and network meta-analysisThe Lancet Gastroenterology & Hepatology, 2021
- LDPolypVideo Benchmark: A Large-Scale Colonoscopy Video Dataset of Diverse Polyps2021
- Development of a computer-aided detection system for colonoscopy and a publicly accessible large colonoscopy video database (with video)Gastrointestinal Endoscopy, 2021
- Computer-aided detection-assisted colonoscopy: classification and relevance of false positivesGastrointestinal Endoscopy, 2020