DNeRV: Modeling Inherent Dynamics via Difference Neural Representation for Videos
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2031-2040
Abstract
Existing implicit neural representation (INR) methods do not fully exploit spatiotemporal redundancies in videos. Index-based INRs ignore the content-specific spatial features and hybrid INRs ignore the contextual dependency on adjacent frames, leading to poor modeling capability for scenes with large motion or dynamics. We analyze this limitation from the perspective of function fitting and reveal the importance of frame difference. To use explicit motion information, we propose Difference Neural Representation for Videos (DNeRV), which consists of two streams for content and frame difference. We also introduce a collaborative content unit for effective feature fusion. We test DNeRV for video compression, inpainting, and interpolation. D-NeRVachieves competitive results against the state-of-the-art neural compression approaches and outperforms existing implicit methods on downstream inpainting and interpolation for $960\times 1920$ videos.
Keywords
Funding Information
- National Key Research and Development Project of China (2022YFF0902402)
This publication has 33 references indexed in Scilit:
- Learning to Forecast and Refine Residual Motion for Image-to-Video Generation2018
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- VMAF reproducibility: Validating a perceptual practical video quality metric2017
- Convolutional Two-Stream Network Fusion for Video Action Recognition2016
- A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation2016
- Overview of the High Efficiency Video Coding (HEVC) StandardIEEE Transactions on Circuits and Systems for Video Technology, 2012
- Decoding by Linear ProgrammingIEEE Transactions on Information Theory, 2005
- Multiscale structural similarity for image quality assessment2004
- Overview of the H.264/AVC video coding standardIEEE Transactions on Circuits and Systems for Video Technology, 2003
- Long Short-Term MemoryNeural Computation, 1997