Learning to Sketch with Shortcut Cycle Consistency
Authors:
- Jifei SongJSJifei Song
- Kaiyue PangKPKaiyue Pang
- Yi-Zhe SongYSYi-Zhe Song
- Tao XiangTXTao Xiang
- Timothy M. HospedalesTHTimothy M. Hospedales
- 1 June 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 801-810
Abstract
To see is to sketch - free-hand sketching naturally builds ties between human and machine vision. In this paper, we present a novel approach for translating an object photo to a sketch, mimicking the human sketching process. This is an extremely challenging task because the photo and sketch domains differ significantly. Furthermore, human sketches exhibit various levels of sophistication and abstraction even when depicting the same object instance in a reference photo. This means that even if photo-sketch pairs are available, they only provide weak supervision signal to learn a translation model. Compared with existing supervised approaches that solve the problem of D(E(photo)) → sketch), where E(·) and D(·) denote encoder and decoder respectively, we take advantage of the inverse problem (e.g., D(E(sketch) → photo), and combine with the unsupervised learning tasks of within-domain reconstruction, all within a multi-task learning framework. Compared with existing unsupervised approaches based on cycle consistency (i.e., D(E(D(E(photo)))) → photo), we introduce a shortcut consistency enforced at the encoder bottleneck (e.g., D(E(photo)) → photo) to exploit the additional self-supervision. Both qualitative and quantitative results show that the proposed model is superior to a number of state-of-the-art alternatives. We also show that the synthetic sketches can be used to train a better fine-grained sketch-based image retrieval (FG-SBIR) model, effectively alleviating the problem of sketch data scarcity.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Synergistic Instance-Level Subspace Alignment for Fine-Grained Sketch-Based Image RetrievalIEEE Transactions on Image Processing, 2017
- Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network2017
- Scribbler: Controlling Deep Image Synthesis with Sketch and Color2017
- Image-to-Image Translation with Conditional Adversarial Networks2017
- Sketch-a-Net: A Deep Neural Network that Beats HumansInternational Journal of Computer Vision, 2016
- Sketch Me That Shoe2016
- Depth-Based Hand Pose Estimation: Data, Methods, and Challenges2015
- Articulated pose estimation with tiny synthetic videos2015
- Real-Time Continuous Pose Recovery of Human Hands Using Convolutional NetworksACM Transactions on Graphics, 2014
- How do humans sketch objects?ACM Transactions on Graphics, 2012