Integrating Scene Text and Visual Appearance for Fine-Grained Image Classification with Convolutional Neural Networks
Authors:
- Xiang BaiXBXiang Bai
- Mingkun YangMYMingkun Yang
- Pengyuan LyuPLPengyuan Lyu
- Yongchao XuYXYongchao Xu
Preprint
Abstract
Text in natural images contains plenty of semantics that are often highly relevant to objects or scene. In this paper, we are concerned with the problem on fully exploiting scene text for visual understanding. The basic idea is combining word representations and deep visual features into a globally trainable deep convolutional neural network. First, the recognized words are obtained by a scene text reading system. Then, we combine the word embedding of the recognized words and the deep visual features into a single representation, which is optimized by a convolutional neural network for fine-grained image classification. In our framework, the attention mechanism is adopted to reveal the relevance between each recognized word and the given image, which reasonably enhances the recognition performance. We have performed experiments on two datasets: Con-Text dataset and Drink Bottle dataset, that are proposed for fine-grained classification of business places and drink bottles respectively. The experimental results consistently demonstrate that the proposed method combining textual and visual cues significantly outperforms classification with only visual representations. Moreover, we have shown that the learned representation improves the retrieval performance on the drink bottle images by a large margin, which is potentially useful in product search.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Attention-Based Multimodal Fusion for Video Description2017
- What is a good evaluation protocol for text localization systems? Concerns, arguments, comparisons and solutionsImage and Vision Computing, 2016
- Scene text detection and recognition: recent advances and future trendsFrontiers of Computer Science, 2015
- Reading Text in the Wild with Convolutional Neural NetworksInternational Journal of Computer Vision, 2015
- Scene text extraction based on edges and support vector regressionInternational Journal on Document Analysis and Recognition (IJDAR), 2015
- A Unified Framework for Multioriented Text Detection and RecognitionIEEE Transactions on Image Processing, 2014
- Multimodal page classification in administrative document image streamsInternational Journal on Document Analysis and Recognition (IJDAR), 2014
- Part-Based R-CNNs for Fine-Grained Category Detection2014
- Text String Detection From Natural Scenes by Structure-Based Partition and GroupingIEEE Transactions on Image Processing, 2011
- A Hybrid Approach to Detect and Localize Texts in Natural Scene ImagesIEEE Transactions on Image Processing, 2010