Enhancing Fine-Grained Sketch-based Image Retrieval through Contextual Information
- 27 February 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. PP (15209210) , 1-12
Abstract
Fine-grained sketch-based image retrieval aims to retrieve matched natural photos at instance level based on hand-drawn sketches. Understanding sketches and aligning them with photos in the joint embedding space are the primary challenges. Previous works have attempted to improve the understanding and alignment of fine-grained semantics by leveraging a selected subset of local features based on the local matching strategy. However, these approaches rely heavily on local features to capture regional visual details, which introduces unavoidable uncertainty and noise in the matching process. In addition, existing works lack the ability to effectively separate unmatched pairs with similar overall semantics in the common space, which may disrupt fine-grained semantic alignment. In this paper, we present CIGSPA: a Contextual Information Guided Sketch-Photo Alignment framework that leverages latent contextual information to enhance sketch understanding and sketch-photo matching without additional knowledge and prior supervision. Specifically, we first design an Adaptive Local Feature Filter module to refine feature similarity based on the context of local features and select more appropriate features, which ensure the semantic fidelity and spatial compactness of salient features. Secondly, we design a Contextual Weighted Contrastive loss to guide global sketch-photo matching by leveraging discriminative contextual information from photos, and hence aligns the sketch and photo at the instance level. Our comprehensive experiments on six public datasets demonstrate the effectiveness of our proposed CIGSPA. The code is available at https://github.com/jinyuxu-whut/CIGSPA4FG-SBIR.Keywords
Funding Information
- National Natural Science Foundation of China (62271360)
- National Key Research and Development Program of China (2024YFF0907002)
This publication has 44 references indexed in Scilit:
- Elevating All Zero-Shot Sketch-Based Image Retrieval Through Multimodal Prompt Learning2024
- Freeview Sketching: View-Aware Fine-Grained Sketch-Based Image Retrieval2024
- Expansion window local alignment weighted network for fine-grained sketch-based image retrievalPattern Recognition, 2023
- Fine-Grained Instance-Level Sketch-Based Image RetrievalInternational Journal of Computer Vision, 2020
- Deep Shape Matching2018
- Sketching out the details: Sketch-based image retrieval using convolutional neural networks with multi-stage regressionComputers & Graphics, 2018
- Triplet Networks Feature Masking for Sketch-Based Image Retrieval2017
- Sketch-a-Net: A Deep Neural Network that Beats HumansInternational Journal of Computer Vision, 2016
- Deep Residual Learning for Image Recognition2016
- Hello neighbor: Accurate object retrieval with k-reciprocal nearest neighbors2011