You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval
Preprint
Abstract
Two primary input modalities prevail in image retrieval: sketch and text. While text is widely used for inter-category retrieval tasks, sketches have been established as the sole preferred modality for fine-grained image retrieval due to their ability to capture intricate visual details. In this paper, we question the reliance on sketches alone for fine-grained image retrieval by simultaneously exploring the fine-grained representation capabilities of both sketch and text, orchestrating a duet between the two. The end result enables precise retrievals previously unattainable, allowing users to pose ever-finer queries and incorporate attributes like colour and contextual cues from text. For this purpose, we introduce a novel compositionality framework, effectively combining sketches and text using pre-trained CLIP models, while eliminating the need for extensive fine-grained textual descriptions. Last but not least, our system extends to novel applications in composed image retrieval, domain attribute transfer, and fine-grained generation, providing solutions for various real-world scenarios.Keywords
All Related Versions
This publication has 49 references indexed in Scilit:
- FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context2022
- FashionViL: Fashion-Focused Vision-and-Language Representation Learning2022
- A Sketch is Worth a Thousand Words: Image Retrieval with Text and Sketch2022
- CoGS: Controllable Generation and Search from Sketch and Style2022
- “This Is My Unicorn, Fluffy”: Personalizing Frozen Vision-Language Representations2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches2020
- Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks2020
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Microsoft COCO: Common Objects in Context2014