Sketchformer: Transformer-Based Representation for Sketched Structure
Authors:
- 1 June 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 14141-14150
Abstract
Sketchformer is a novel transformer-based representation for encoding free-hand sketches input in a vector form, i.e. as a sequence of strokes. Sketchformer effectively addresses multiple tasks: sketch classification, sketch based image retrieval (SBIR), and the reconstruction and interpolation of sketches. We report several variants exploring continuous and tokenized input representations, and contrast their performance. Our learned embedding, driven by a dictionary learning tokenization scheme, yields state of the art performance in classification and image retrieval tasks, when compared against baseline representations driven by LSTM sequence to sequence architectures: SketchRNN and derivatives. We show that sketch reconstruction and interpolation are improved significantly by the Sketchformer embedding for complex sketches with longer stroke sequences.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- Sketch-based image retrieval via Siamese convolutional neural network2016
- SketchNet: Sketch Classification with Web Images2016
- DeepSketch 2: Deep convolutional neural networks for partial sketch recognition2016
- Scalable Sketch-Based Image Retrieval Using Color Gradient Features2015
- Sketch classification and classification-driven analysis using Fisher vectorsACM Transactions on Graphics, 2014
- Learning Fine-Grained Image Similarity with Deep Ranking2014
- A performance evaluation of gradient field HOG descriptor for sketch based image retrievalComputer Vision and Image Understanding, 2013
- How do humans sketch objects?ACM Transactions on Graphics, 2012
- MindFinder2010
- ALGORITHMS FOR THE REDUCTION OF THE NUMBER OF POINTS REQUIRED TO REPRESENT A DIGITIZED LINE OR ITS CARICATURECartographica, 1973