Multimodal Convolutional Neural Networks for Matching Image and Sentence
Preprint
Abstract
In this paper, we propose multimodal convolutional neural networks (m-CNNs) for matching image and sentence. Our m-CNN provides an end-to-end framework with convolutional architectures to exploit image representation, word composition, and the matching relations between the two modalities. More specifically, it consists of one image CNN encoding the image content, and one matching CNN learning the joint representation of image and sentence. The matching CNN composes words to different semantic fragments and learns the inter-modal relations between image and the composed fragments at different levels, thus fully exploit the matching relations between image and sentence. Experimental results on benchmark databases of bidirectional image and sentence retrieval demonstrate that the proposed m-CNNs can effectively capture the information necessary for image and sentence matching. Specifically, our proposed m-CNNs for bidirectional image and sentence retrieval on Flickr30K and Microsoft COCO databases achieve the state-of-the-art performances.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- Going deeper with convolutions2015
- Show and tell: A neural image caption generator2015
- Deep correlation for matching images and text2015
- Associating neural word embeddings with deep image representations using Fisher Vectors2015
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- Microsoft COCO: Common Objects in Context2014
- Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition2014
- Improving Image-Sentence Embeddings Using Large Weakly Annotated Photo Collections2014
- Applying Convolutional Neural Networks concepts to hybrid NN-HMM model for speech recognition2012
- Recognition using visual phrases2011