An Extensive Study on Cross-Dataset Bias and Evaluation Metrics Interpretation for Machine Learning Applied to Gastrointestinal Tract Abnormality Classification
Authors:
Open Access
- 22 June 2020
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Computing for Healthcare
- Vol. 1 (3) , 1-29
Abstract
Precise and efficient automated identification of gastrointestinal (GI) tract diseases can help doctors treat more patients and improve the rate of disease detection and identification. Currently, automatic analysis of diseases in the GI tract is a hot topic in both computer science and medical-related journals. Nevertheless, the evaluation of such an automatic analysis is often incomplete or simply wrong. Algorithms are often only tested on small and biased datasets, and cross-dataset evaluations are rarely performed. A clear understanding of evaluation metrics and machine learning models with cross datasets is crucial to bring research in the field to a new quality level. Toward this goal, we present comprehensive evaluations of five distinct machine learning models using global features and deep neural networks that can classify 16 different key types of GI tract conditions, including pathological findings, anatomical landmarks, polyp removal conditions, and normal findings from images captured by common GI tract examination instruments. In our evaluation, we introduce performance hexagons using six performance metrics, such as recall, precision, specificity, accuracy, F1-score, and the Matthews correlation coefficient to demonstrate how to determine the real capabilities of models rather than evaluating them shallowly. Furthermore, we perform cross-dataset evaluations using different datasets for training and testing. With these cross-dataset evaluations, we demonstrate the challenge of actually building a generalizable model that could be used across different hospitals. Our experiments clearly show that more sophisticated performance metrics and evaluation methods need to be applied to get reliable models rather than depending on evaluations of the splits of the same dataset—that is, the performance metrics should always be interpreted together rather than relying on a single metric.Keywords
All Related Versions
Funding Information
- Norges Forskningsråd (263248)
This publication has 32 references indexed in Scilit:
- Part-Based Multiderivative Edge Cross-Sectional Profiles for Polyp Detection in ColonoscopyIEEE Journal of Biomedical and Health Informatics, 2014
- European guidelines for quality assurance in colorectal cancer screening and diagnosis. First Edition – Executive summaryEndoscopy, 2012
- Towards automatic polyp detection with a polyp appearance modelPattern Recognition, 2012
- Factors influencing the miss rate of polyps in a back-to-back colonoscopy studyEndoscopy, 2012
- The WEKA data mining softwareACM SIGKDD Explorations Newsletter, 2009
- Logistic Model TreesMachine Learning, 2005
- A Tutorial on the Cross-Entropy MethodAnnals of Operations Research, 2005
- Computer-aided tumor detection in endoscopic video using color wavelet featuresIEEE Transactions on Information Technology in Biomedicine, 2003
- Additive logistic regression: a statistical view of boosting (With discussion and a rejoinder by the authors)The Annals of Statistics, 2000
- C4.5: Programs for Machine Learning by J. Ross Quinlan. Morgan Kaufmann Publishers, Inc., 1993Machine Learning, 1994