Concurrent Video Denoising and Deblurring for Dynamic Scenes
Open Access
- 19 November 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 9, 157437-157446
Abstract
Dynamic scene video deblurring is a challenging task due to the spatially variant blur inflicted by independently moving objects and camera shakes. Recent deep learning works bypass the ill-posedness of explicitly deriving the blur kernel by learning pixel-to-pixel mappings, which is commonly enhanced by larger region awareness. This is a difficult yet simplified scenario because noise is neglected when it is omnipresent in a wide spectrum of video processing applications. Despite its relevance, the problem of concurrent noise and dynamic blur has not yet been addressed in the deep learning literature. To this end, we analyze existing state-of-the-art deblurring methods and encounter their limitations in handling non-uniform blur under strong noise conditions. Thereafter, we propose a first-to-date work that addresses blur- and noise-free frame recovery by casting the restoration problem into a multi-task learning framework. Our contribution is threefold: a) We propose R2-D4, a multi-scale encoder architecture attached to two cascaded decoders performing the restoration task in two steps. b) We design multi-scale residual dense modules, bolstered by our modulated efficient channel attention, to enhance the encoder representations via augmenting deformable convolutions to capture longer-range and object-specific context that assists blur kernel estimation under strong noise. c) We perform extensive experiments and evaluate state-of-the-art approaches on a publicly available dataset under different noise levels. The proposed method performs favorably under all noise levels while retaining a reasonably low computational and memory footprint.Keywords
Funding Information
- National Centre for Research and Development, Poland (POIR.01.01.01-00-0076/19)
This publication has 42 references indexed in Scilit:
- Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring2017
- Deep Video Deblurring for Hand-Held Cameras2017
- Blind Space-Variant Single-Image Restoration of Defocus Blur2017
- The Little Engine That Could: Regularization by Denoising (RED)SIAM Journal on Imaging Sciences, 2017
- Blind Deblurring and Denoising of Images Corrupted by Unidirectional Object Motion Blur and Sensor NoiseIEEE Transactions on Image Processing, 2016
- Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network2016
- A Convex Approach for Image Restoration with Exact Poisson--Gaussian LikelihoodSIAM Journal on Imaging Sciences, 2015
- Space Variant Blind Image RestorationSIAM Journal on Imaging Sciences, 2014
- Unnatural L0 Sparse Representation for Natural Image Deblurring2013
- Video Denoising, Deblocking, and Enhancement Through Separable 4-D Nonlocal Spatiotemporal TransformsIEEE Transactions on Image Processing, 2012