Breathing Life Into Sketches Using Text-to-Video Priors
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4325-4336
Abstract
A sketch is one of the most intuitive and versatile tools humans use to convey their ideas visually. An animated sketch opens another dimension to the expression of ideas and is widely used by designers for a variety of purposes. Animating sketches is a laborious process, requiring extensive experience and professional design skills. In this work, we present a method that automatically adds motion to a single-subject sketch (hence, “breathing life into it”), merely by providing a text prompt indicating the desired motion. The output is a short animation provided in vector representation, which can be easily edited. Our method does not require extensive training, but instead leverages the motion prior of a large pretrained text-to- video diffusion model using a score-distillation loss to guide the placement of strokes. To promote natural and smooth motion and to better preserve the sketch's appearance, we model the learned motion through two components. The first governs small local deformations and the second controls global affine transformations. Surprisingly, wefind that even models that struggle to generate sketch videos on their own can still serve as a useful backbone for animating abstract representations.Keywords
All Related Versions
This publication has 56 references indexed in Scilit:
- Drawing as a versatile cognitive toolNature Reviews Psychology, 2023
- Expanding Language-Image Pretrained Models for General Video Recognition2022
- DoodleFormer: Creative Sketch Drawing with Transformers2022
- NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion2022
- TiVGAN: Text to Image to Video Generation With Step-by-Step Evolutionary GeneratorIEEE Access, 2020
- Sketch-BERT: Learning Sketch Bidirectional Encoder Representation From Transformers by Self-Supervised Learning of Sketch Gestalt2020
- Photo Wake-Up: 3D Character Animation From a Single Photo2019
- Learning to Sketch with Shortcut Cycle Consistency2018
- Pleistocene cave art from Sulawesi, IndonesiaNature, 2014
- Sketch-Based Skeleton-Driven 2D Animation and Motion Capture2011