SketchFusion: Learning Universal Sketch Features through Fusing Foundation Models
Preprint
Abstract
While foundation models have revolutionised computer vision, their effectiveness for sketch understanding remains limited by the unique challenges of abstract, sparse visual inputs. Through systematic analysis, we uncover two fundamental limitations: Stable Diffusion (SD) struggles to extract meaningful features from abstract sketches (unlike its success with photos), and exhibits a pronounced frequency-domain bias that suppresses essential low-frequency components needed for sketch understanding. Rather than costly retraining, we address these limitations by strategically combining SD with CLIP, whose strong semantic understanding naturally compensates for SD's spatial-frequency biases. By dynamically injecting CLIP features into SD's denoising process and adaptively aggregating features across semantic levels, our method achieves state-of-the-art performance in sketch retrieval (+3.35%), recognition (+1.06%), segmentation (+29.42%), and correspondence learning (+21.22%), demonstrating the first truly universal sketch feature representation in the era of foundation models.Keywords
All Related Versions
This publication has 72 references indexed in Scilit:
- Boost Your NeRF: A Model-Agnostic Mixture of Experts Framework for High Quality and Efficient Rendering2024
- A Sketch is Worth a Thousand Words: Image Retrieval with Text and Sketch2022
- A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-Language Model2022
- CoGS: Controllable Generation and Search from Sketch and Style2022
- Garment Ideation: Iterative View-Aware Sketch-Based Garment Modeling2022
- Structure-Aware 3D VR Sketch to 3D Shape Retrieval2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Deep Sketch Hashing: Fast Free-Hand Sketch-Based Image Retrieval2017
- Deep Residual Learning for Image Recognition2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015