CoSMo: Content-Style Modulation for Image Retrieval with Text Feedback
Authors:
- 1 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 802-812
Abstract
We tackle the task of image retrieval with text feedback, where a reference image and modifier text are combined to identify the desired target image. We focus on designing an image-text compositor, i.e., integrating multi-modal inputs to produce a representation similar to that of the target image. In our algorithm, Content-Style Modulation (CoSMo), we approach this challenge by introducing two modules based on deep neural networks: the content and style modulators. The content modulator performs local updates to the reference image feature after normalizing the style of the image, where a disentangled multi-modal non-local block is employed to achieve the desired content modifications. Then, the style modulator reintroduces global style information to the updated feature. We provide an in-depth view of our algorithm and its design choices, and show that it accomplishes outstanding performance on multiple image-text retrieval benchmarks. Our code can be found at: https://github.com/postBG/CosMo.pytorchKeywords
Funding Information
- Samsung
This publication has 46 references indexed in Scilit:
- Spatial-Semantic Image Search by Visual Feature Synthesis2017
- Image Style Transfer Using Convolutional Neural Networks2016
- The sketchy databaseACM Transactions on Graphics, 2016
- Sketch Me That Shoe2016
- Deep Residual Learning for Image Recognition2016
- Image Question Answering Using Convolutional Neural Network with Dynamic Parameter Prediction2016
- DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations2016
- Deep visual-semantic alignments for generating image descriptions2015
- Show and tell: A neural image caption generator2015
- Learning a Similarity Metric Discriminatively, with Application to Face Verification2005