Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning
Authors:
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3605-3615
Abstract
Most methods for conditional video synthesis use a single modality as the condition. This comes with major limitations. For example, it is problematic for a model conditioned on an image to generate a specific motion trajectory desired by the user since there is no means to provide motion information. Conversely, language information can describe the desired motion, while not precisely defining the content of the video. This work presents a multimodal video generation framework that benefits from text and images provided jointly or separately. We leverage the recent progress in quantized representations for videos and apply a bidirectional transformer with multiple modalities as inputs to predict a discrete video representation. To improve video quality and consistency, we propose a new video token trained with self-learning and an improved mask-prediction algorithm for sampling video tokens. We introduce text augmentation to improve the robustness of the textual representation and diversity of generated videos. Our framework can incorporate various visual modalities, such as segmentation masks, drawings, and partially occluded images. It can generate much longer sequences than the one used for training. In addition, our model can extract visual information as suggested by the text prompt, e.g., “an object in image one is moving northeast”, and generate corresponding videos. We run evaluations on three public datasets and a newly collected dataset labeled with facial attributes, achieving state-of-the-art generation results on all four 1 1 Code: https://github.com/snap-research/MMVID and Webpage..Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- Voxceleb: Large-scale speaker verification in the wildComputer Speech & Language, 2020
- Conditional GAN with Discriminative Filter Generation for Text-to-Video Synthesis2019
- Video Generation From Single Semantic Label Map2019
- Mask-Predict: Parallel Decoding of Conditional Masked Language Models2019
- MoCoGAN: Decomposing Motion and Content for Video Generation2018
- To Create What You Tell2017
- Temporal Generative Adversarial Nets with Singular Value Clipping2017
- The Pose Knows: Video Forecasting by Generating Pose Futures2017
- Attentive Semantic Video Generation Using Captions2017
- Deep Learning Face Attributes in the Wild2015