Effective conditioned and composed image retrieval combining CLIP-based features
Authors:
Top Cited Papers
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 21434-21442
Abstract
Conditioned and composed image retrieval extend CBIR systems by combining a query image with an additional text that expresses the intent of the user, describing additional requests w.r.t. the visual content of the query image. This type of search is interesting for e-commerce applications, e.g. to develop interactive multimodal searches and chat-bots. In this demo, we present an interactive system based on a combiner network, trained using contrastive learning, that combines visual and textual features obtained from the OpenAI CLIP network to address conditioned CBIR. The system can be used to improve e-shop search engines. For example, considering the fashion domain it lets users search for dresses, shirts and toptees using a candidate start image and expressing some visual differences w.r.t. its visual con-tent, e.g. asking to change color, pattern or shape. The pro-posed network obtains state-of-the-art performance on the FashionIQ dataset and on the more recent CIRR dataset, showing its applicability to the fashion domain for conditioned retrieval, and to more generic content considering the more general task of composed image retrieval.Keywords
This publication has 29 references indexed in Scilit:
- Learning a Unified Embedding for Visual Search at Pinterest2019
- DeepStyle: Multimodal Search Engine for Fashion and Interior DesignIEEE Access, 2019
- How to Read Paintings: Semantic Art Understanding with Multi-modal Retrieval2019
- A Corpus for Reasoning about Natural Language Grounded in Photographs2019
- Visual Search at Alibaba2018
- Learning Fashion Compatibility with Bidirectional LSTMs2017
- Deep Artwork Detection and Retrieval for Automatic Context-Aware Audio GuidesACM Transactions on Multimedia Computing, Communications, and Applications, 2017
- SIFT Meets CNN: A Decade Survey of Instance RetrievalIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Deep Residual Learning for Image Recognition2016
- Content-based image retrieval at the end of the early yearsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2000