Erase or Fill? Deep Joint Recurrent Rain Removal and Reconstruction in Videos
Authors:
- Jiaying LiuJLJiaying Liu
- Wenhan YangWYWenhan Yang
- Shuai YangSYShuai Yang
- Zongming GuoZGZongming Guo
- 1 June 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3233-3242
Abstract
In this paper, we address the problem of video rain removal by constructing deep recurrent convolutional networks. We visit the rain removal case by considering rain occlusion regions, i.e. the light transmittance of rain streaks is low. Different from additive rain streaks, in such rain occlusion regions, the details of background images are completely lost. Therefore, we propose a hybrid rain model to depict both rain streaks and occlusions. With the wealth of temporal redundancy, we build a Joint Recurrent Rain Removal and Reconstruction Network (J4R-Net) that seamlessly integrates rain degradation classification, spatial texture appearances based rain removal and temporal coherence based background details reconstruction. The rain degradation classification provides a binary map that reveals whether a location is degraded by linear additive streaks or occlusions. With this side information, the gate of the recurrent unit learns to make a trade-off between rain streak removal and background details reconstruction. Extensive experiments on a series of synthetic and real videos with rain streaks verify the superiority of the proposed method over previous state-of-the-art methods.Keywords
This publication has 30 references indexed in Scilit:
- A Rain Pixel Recovery Algorithm for Videos With Highly Dynamic ScenesIEEE Transactions on Image Processing, 2013
- Video post processing: low-latency spatiotemporal approach for detection and removal of rainIET Image Processing, 2012
- Rain or Snow Detection in Image Sequences Through Use of a Histogram of Orientation of StreaksInternational Journal of Computer Vision, 2011
- A Probabilistic Approach for Detection and Removal of Rain from VideosIETE Journal of Research, 2011
- Analysis of Rain and Snow in Frequency SpaceInternational Journal of Computer Vision, 2009
- Pixel Based Temporal Analysis Using Chromatic Property for Removing Rain from VideosComputer and Information Science, 2009
- Scope of validity of PSNR in image/video quality assessmentElectronics Letters, 2008
- Vision and RainInternational Journal of Computer Vision, 2007
- Photorealistic rendering of rain streaksACM Transactions on Graphics, 2006
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004