Target-Oriented Deformation of Visual-Semantic Embedding Space
- 1 January 2021
- journal article
- research article
- Published by Institute of Electronics, Information and Communications Engineers (IEICE) in IEICE Transactions on Information and Systems
- Vol. E104.D (1) , 24-33
Abstract
Multimodal embedding is a crucial research topic for cross-modal understanding, data mining, and translation. Many studies have attempted to extract representations from given entities and align them in a shared embedding space. However, because entities in different modalities exhibit different abstraction levels and modality-specific information, it is insufficient to embed related entities close to each other. In this study, we propose the Target-Oriented Deformation Network (TOD-Net), a novel module that continuously deforms the embedding space into a new space under a given condition, thereby providing conditional similarities between entities. Unlike methods based on cross-modal attention applied to words and cropped images, TOD-Net is a post-process applied to the embedding space learned by existing embedding systems and improves their performances of retrieval. In particular, when combined with cutting-edge models, TOD-Net gains the state-of-the-art image-caption retrieval model associated with the MS COCO and Flickr30k datasets. Qualitative analysis reveals that TOD-Net successfully emphasizes entity-specific concepts and retrieves diverse targets via handling higher levels of diversity than existing models.Keywords
All Related Versions
This publication has 29 references indexed in Scilit:
- Conditional Similarity Networks2017
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence ModelsInternational Journal of Computer Vision, 2016
- Identity Mappings in Deep Residual Networks2016
- WELDON: Weakly Supervised Learning of Deep Convolutional Neural Networks2016
- Learning Deep Structure-Preserving Image-Text Embeddings2016
- Deep Residual Learning for Image Recognition2016
- Multimodal Convolutional Neural Networks for Matching Image and Sentence2015
- Microsoft COCO: Common Objects in Context2014
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- Approximation by superpositions of a sigmoidal functionMathematics of Control, Signals, and Systems, 1989