Improving Temporal Stability and Accuracy for Endoscopic Video Tissue Classification Using Recurrent Neural Networks
Open Access
Abstract
Early Barrett’s neoplasia are often missed due to subtle visual features and inexperience of the non-expert endoscopist with such lesions. While promising results have been reported on the automated detection of this type of early cancer in still endoscopic images, video-based detection using the temporal domain is still open. The temporally stable nature of video data in endoscopic examinations enables to develop a framework that can diagnose the imaged tissue class over time, thereby yielding a more robust and improved model for spatial predictions. We show that the introduction of Recurrent Neural Network nodes offers a more stable and accurate model for tissue classification, compared to classification on individual images. We have developed a customized Resnet18 feature extractor with four types of classifiers: Fully Connected (FC), Fully Connected with an averaging filter (FC Avg (n = 5)), Long Short Term Memory (LSTM) and a Gated Recurrent Unit (GRU). Experimental results are based on 82 pullback videos of the esophagus with 46 high-grade dysplasia patients. Our results demonstrate that the LSTM classifier outperforms the FC, FC Avg (n = 5) and GRU classifier with an average accuracy of 85.9% compared to 82.2%, 83.0% and 85.6%, respectively. The benefit of our novel implementation for endoscopic tissue classification is the inclusion of spatio-temporal information for improved and robust decision making, and it is the first step towards full temporal learning of esophageal cancer detection in endoscopic video.Keywords
This publication has 19 references indexed in Scilit:
- Review on the Applications of Deep Learning in the Analysis of Gastrointestinal Endoscopy ImagesIEEE Access, 2019
- A survey of feature extraction and fusion of deep learning for detection of abnormalities in video endoscopy of gastrointestinal-tractArtificial Intelligence Review, 2019
- Artificial Intelligence-Based Classification of Multiple Gastrointestinal Diseases Using Endoscopy Videos for Clinical DiagnosisJournal of Clinical Medicine, 2019
- The Argos project: The development of a computer‐aided detection system to improve detection of Barrett's neoplasia on white light endoscopyUnited European Gastroenterology Journal, 2019
- Action Recognition with 3D ConvNet-GRU Architecture2018
- Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning modelGut, 2017
- Integrating Online and Offline Three-Dimensional Deep Learning for Automated Polyp Detection in Colonoscopy VideosIEEE Journal of Biomedical and Health Informatics, 2016
- Scalable gastroscopic video summarization via similar-inhibition dictionary selectionArtificial Intelligence in Medicine, 2016
- Optical Deformability as an Inherent Cell Marker for Testing Malignant Transformation and Metastatic CompetenceBiophysical Journal, 2005
- Individual Comparisons by Ranking MethodsBiometrics Bulletin, 1945