Video Frame Prediction by Deep Multi-Branch Mask Network
Authors:
- 1 April 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 31 (4) , 1283-1295
Abstract
Future frame prediction in video is one of the most important problem in computer vision, and useful for a range of practical applications, such as intention prediction or video anomaly detection. However, this task is challenging because of the complex and dynamic evolution of scene. The difficulty of video frame prediction is to model the inherent spatio-temporal correlation between frames and pose an adaptive and flexible framework for large motion change or appearance variation. In this paper, we construct a deep multi-branch mask network (DMMNet) which adaptively fuses the advantages of optical flow warping and RGB pixel synthesizing methods, i.e., the common two kinds of approaches in this task. In the procedure of DMMNet, we add mask layer in each branch to adaptively adjust the magnitude range of estimated optical flow and the weight of predicted frames by optical flow warping and RGB pixel synthesizing, respectively. In other words, we provide a more flexible masking network for motion and appearance fusion on video frame prediction. Exhaustive experiments on Caltech pedestrian and UCF101 datasets show that the proposed model can obtain favorable video frame prediction performance compared with the state-of-the-art methods. In addition, we also put our model into the video anomaly detection problem, and the superiority is verified by the experiments on UCSD dataset.Keywords
Funding Information
- National Natural Science Foundation of China (61751308, 61773311, 61603057)
- Fundamental Research Funds for the Central Universities, CHD (300102320202)
This publication has 32 references indexed in Scilit:
- Learning Temporal Regularity in Video Sequences2016
- EpicFlow: Edge-preserving interpolation of correspondences for optical flow2015
- Fully convolutional networks for semantic segmentation2015
- Learning to generate chairs with convolutional neural networks2015
- Multiview Plus Depth Video Coding With Temporal Prediction View SynthesisIEEE Transactions on Circuits and Systems for Video Technology, 2015
- Online Anomaly Detection in Crowd Scenes via Structure AnalysisIEEE Transactions on Cybernetics, 2014
- Learning-based approach for online lane change intention prediction2013
- Probabilistic movement modeling for intention inference in human–robot interactionThe International Journal of Robotics Research, 2013
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004
- Long Short-Term MemoryNeural Computation, 1997