Semantically Tied Paired Cycle Consistency for Any-Shot Sketch-Based Image Retrieval
Open Access
- 29 July 2020
- journal article
- research article
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 128 (10-11) , 2684-2703
Abstract
Low-shot sketch-based image retrieval is an emerging task in computer vision, allowing to retrieve natural images relevant to hand-drawn sketch queries that are rarely seen during the training phase. Related prior works either require aligned sketch-image pairs that are costly to obtain or inefficient memory fusion layer for mapping the visual information to a semantic space. In this paper, we address any-shot,i.e. zero-shot and few-shot, sketch-based image retrieval (SBIR) tasks, where we introduce the few-shot setting for SBIR. For solving these tasks, we propose a semantically aligned paired cycle-consistent generative adversarial network (SEM-PCYC) for any-shot SBIR, where each branch of the generative adversarial network maps the visual information from sketch and image to a common semantic space via adversarial training. Each of these branches maintains cycle consistency that only requires supervision at the category level, and avoids the need of aligned sketch-image pairs. A classification criteria on the generators’ outputs ensures the visual to semantic space mapping to be class-specific. Furthermore, we propose to combine textual and hierarchical side information via an auto-encoder that selects discriminating side information within a same end-to-end model. Our results demonstrate a significant boost in any-shot SBIR performance over the state-of-the-art on the extended version of the challenging Sketchy, TU-Berlin and QuickDraw datasets.Keywords
Funding Information
- European Union (665919)
- Deutsche Forschungsgemeinschaft (390727645)
- Ministerio de Ciencia, Innovación y Universidades (RTI2018-102285-A-I00)
This publication has 65 references indexed in Scilit:
- Label-Embedding for Image ClassificationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Bit-Scalable Deep Hashing With Regularized Similarity Learning for Image Retrieval and Person Re-IdentificationIEEE Transactions on Image Processing, 2015
- Evaluation of output embeddings for fine-grained image classification2015
- Community detection in social networksProceedings of the VLDB Endowment, 2015
- COSTA: Co-Occurrence Statistics for Zero-Shot Classification2014
- Glove: Global Vectors for Word Representation2014
- Intra-category sketch-based image retrieval by matching deformable part models2014
- Attribute-Based Classification for Zero-Shot Visual Object CategorizationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Learning a Similarity Metric Discriminatively, with Application to Face Verification2005
- WordNetCommunications of the ACM, 1995