Composed Image Retrieval via Explicit Erasure and Replenishment With Semantic Alignment
- 12 September 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 31 (10577149) , 5976-5988
Abstract
Composed image retrieval aims at retrieving the desired images, given a reference image and a text piece. To handle this task, two important subprocesses should be modeled reasonably. One is to erase irrelated details of the reference image against the text piece, and the other is to replenish the desired details in the image against the text piece. Nowadays, the existing methods neglect to distinguish between the two subprocesses and implicitly put them together to solve the composed image retrieval task. To explicitly and orderly model the two subprocesses of the task, we propose a novel composed image retrieval method which contains three key components, i.e., Multi-semantic Dynamic Suppression module (MDS), Text-semantic Complementary Selection module (TCS), and Semantic Space Alignment constraints (SSA). Concretely, MDS is to erase irrelated details of the reference image by suppressing its semantic features. TCS aims to select and enhance the semantic features of the text piece and then replenish them to the reference image. In the end, to facilitate the erasure and replenishment subprocesses, SSA aligns the semantics of the two modality features in the final space. Extensive experiments on three benchmark datasets (Shoes, FashionIQ, and Fashion200K) show the superior performance of our approach against state-of-the-art methods.Keywords
Funding Information
- National Key Research and Development Program of China (2021ZD0112100)
- National NSF of China (61972022, U1936212, 62120106009)
This publication has 40 references indexed in Scilit:
- Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering2017
- Spatial-Semantic Image Search by Visual Feature Synthesis2017
- Deep Relative Attributes2017
- Deep Image Retrieval: Learning Global Representations for Image Search2016
- Sketch Me That Shoe2016
- Deep Residual Learning for Image Recognition2016
- Deep Learning Face Representation from Predicting 10,000 Classes2014
- Glove: Global Vectors for Word Representation2014
- Automatic Attribute Discovery and Characterization from Noisy Web Data2010
- Relevance feedback: a power tool for interactive content-based image retrievalIEEE Transactions on Circuits and Systems for Video Technology, 1998