Multimodal Pivots for Image Caption Translation
Preprint
Abstract
We present an approach to improve statistical machine translation of image descriptions by multimodal pivots defined in visual space. Image similarity is computed by a convolutional neural network and incorporated into a target-side translation memory retrieval model where descriptions of most similar images are used to rerank translation outputs. Our approach does not depend on the availability of in-domain parallel data and achieves improvements of 1.4 BLEU over strong baselines.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Fast R-CNN2015
- A Survey of Current Datasets for Vision and Language Research2015
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- Microsoft COCO: Common Objects in Context2014
- BabyTalk: Understanding and Generating Simple Image DescriptionsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Margin Infused Relaxed Algorithm for MosesThe Prague Bulletin of Mathematical Linguistics, 2011
- Using a maximum entropy model to build segmentation lattices for MT2009
- Hierarchical Phrase-Based TranslationComputational Linguistics, 2007