Learning Semantic and Visual Similarity for Endomicroscopy Video Retrieval
- 16 February 2012
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 31 (6) , 1276-1288
Abstract
Content-based image retrieval (CBIR) is a valuable computer vision technique which is increasingly being applied in the medical community for diagnosis support. However, traditional CBIR systems only deliver visual outputs, i.e., images having a similar appearance to the query, which is not directly interpretable by the physicians. Our objective is to provide a system for endomicroscopy video retrieval which delivers both visual and semantic outputs that are consistent with each other. In a previous study, we developed an adapted bag-of-visual-words method for endomicroscopy retrieval, called “Dense-Sift,” that computes a visual signature for each video. In this paper, we present a novel approach to complement visual similarity learning with semantic knowledge extraction, in the field of in vivo endomicroscopy. We first leverage a semantic ground truth based on eight binary concepts, in order to transform these visual signatures into semantic signatures that reflect how much the presence of each semantic concept is expressed by the visual words describing the videos. Using cross-validation, we demonstrate that, in terms of semantic detection, our intuitive Fisher-based method transforming visual-word histograms into semantic estimations outperforms support vector machine (SVM) methods with statistical significance. In a second step, we propose to improve retrieval relevance by learning an adjusted similarity distance from a perceived similarity ground truth. As a result, our distance learning method allows to statistically improve the correlation with the perceived similarity. We also demonstrate that, in terms of perceived similarity, the recall performance of the semantic signatures is close to that of visual signatures and significantly better than those of several state-of-the-art CBIR methods. The semantic signatures are thus able to communicate high-level medical knowledge while being consistent with the low-level visual signatures and much shorter than them. In our resulting retrieval system, we decide to use visual signatures for perceived similarity learning and retrieval, and semantic signatures for the output of an additional information, expressed in the endoscopist own language, which provides a relevant semantic translation of the visual retrieval outputs.Keywords
This publication has 24 references indexed in Scilit:
- Su1493 Comparison Between Video and Mosaic Viewing Mode of Confocal Laser Endomicroscopy (pCLE) in Patients With Barrett's EsophagusGastrointestinal Endoscopy, 2011
- Content-Based Image Retrieval in Radiology: Current Status and Future DirectionsJournal of Digital Imaging, 2010
- An Image Retrieval Approach to Setup Difficulty Levels in Training Systems for Endomicroscopy DiagnosisLecture Notes in Computer Science, 2010
- Robust mosaicing with correction of motion distortions and tissue deformations for in vivo fibered microscopyMedical Image Analysis, 2006
- Local Features and Kernels for Classification of Texture and Object Categories: A Comprehensive StudyInternational Journal of Computer Vision, 2006
- Confocal laser endoscopy for diagnosing intraepithelial neoplasias and colorectal cancer in vivoGastroenterology, 2004
- Representing and Recognizing the Visual Appearance of Materials using Three-dimensional TextonsInternational Journal of Computer Vision, 2001
- Modeling the Shape of the Scene: A Holistic Representation of the Spatial EnvelopeInternational Journal of Computer Vision, 2001
- Comparing correlated correlation coefficients.Psychological Bulletin, 1992
- Statistical and structural approaches to textureProceedings of the IEEE, 1979