Text-Driven Video Prediction
Open Access
- 23 September 2024
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 20 (9) , 1-15
Abstract
Current video generation models usually convert signals indicating appearance and motion received from inputs (e.g., image and text) or latent spaces (e.g., noise vectors) into consecutive frames, fulfilling a stochastic generation process for the uncertainty introduced by latent code sampling. However, this generation pattern lacks deterministic constraints for both appearance and motion, leading to uncontrollable and undesirable outcomes. To this end, we propose a new task called Text-driven Video Prediction (TVP). Taking the first frame and text caption as inputs, this task aims to synthesize the following frames. Specifically, appearance and motion components are provided by the image and caption separately. The key to addressing the TVP task depends on fully exploring the underlying motion information in text descriptions, thus facilitating plausible video generation. In fact, this task is intrinsically a cause-and-effect problem, as the text content directly influences the motion changes of frames. To investigate the capability of text in causal inference for progressive motion information, our TVP framework contains a Text Inference Module (TIM), producing step-wise embeddings to regulate motion inference for subsequent frames. In particular, a refinement mechanism incorporating global motion semantics guarantees coherent generation. Extensive experiments are conducted on Something-Something V2 and Single Moving MNIST datasets. Experimental results demonstrate that our model achieves better results over other baselines, verifying the effectiveness of the proposed framework.Keywords
All Related Versions
Funding Information
- NSFC (62032006)
- Shanghai Science and Technology Program (21JC1400600)
This publication has 32 references indexed in Scilit:
- Music2Dance: DanceNet for Music-Driven Dance GenerationACM Transactions on Multimedia Computing, Communications, and Applications, 2022
- Conditional LSTM-GAN for Melody Generation from LyricsACM Transactions on Multimedia Computing, Communications, and Applications, 2021
- MirrorGAN: Learning Text-To-Image Generation by Redescription2019
- A Style-Based Generator Architecture for Generative Adversarial Networks2019
- Show, Reward, and TellACM Transactions on Multimedia Computing, Communications, and Applications, 2019
- MoCoGAN: Decomposing Motion and Content for Video Generation2018
- Hierarchical Video Generation From Orthogonal Information: Optical Flow and TextureProceedings of the AAAI Conference on Artificial Intelligence, 2018
- Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization2017
- The “Something Something” Video Database for Learning and Evaluating Visual Common Sense2017
- Image-to-Image Translation with Conditional Adversarial Networks2017