WAFP-Net: Weighted Attention Fusion Based Progressive Residual Learning for Depth Map Super-Resolution
- 7 October 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 24 (15209210) , 4113-4127
Abstract
Recently, the development of 3d depth technology brought in many real-world multimedia applications, however, how to tackle real-world degradations in low-resolution (LR) depth maps remains a major challenge, though remarkable progresses have been achieved with DCNN based depth super-resolution (DSR) approaches. Existing DSR models are usually trained and tested on synthetic dataset, which are restrictive and not effective in generalizing to the real-world DSR tasks. In this paper, we aim at alleviating the real-world degradations of different depth sensors in two aspects. First, we classify the generation of LR depth maps into two types: non-linear down-sampling with noise and interval down-sampling, for which different DSR models are learned correspondingly. Second, a novel framework is proposed to handle these two types of LR depth maps in DSR, which consists of four modules: 1) An progressive residual learning module with deep supervision is proposed to learn effective high-frequency components of depth maps in a coarse-to-fine manner. 2) A weighted attention fusion strategy is utilized to intensify the features with abundant high-frequency components in both global and local manners. 3) A multi-stage fusion module is utilized to sufficiently re-exploit the information in the progressive process. 4) A depth refinement module is proposed to improve the depth map by Total Generalized Variation (TGV) regularization and input loss. Extensive experiments on benchmarking datasets demonstrate the superiority of our method over current state-of-the-art DSR methods.Keywords
Funding Information
- Robotics and Autonomous Driving Laboratory of Baidu Research
- National Key Research and Development Program of China (2018AAA0102803)
- National Natural Science Foundation of China (61871325, 61671387)
This publication has 68 references indexed in Scilit:
- A Study of Multimodal Addressee Detection in Human-Human-Computer InteractionIEEE Transactions on Multimedia, 2015
- High-Resolution Stereo Datasets with Subpixel-Accurate Ground Truth2014
- Learning Joint Intensity-Depth Sparse RepresentationsIEEE Transactions on Image Processing, 2014
- Parsing the Hand in Depth ImagesIEEE Transactions on Multimedia, 2014
- Patch Based Synthesis for Single Depth Image Super-Resolution2012
- A Naturalistic Open Source Movie for Optical Flow Evaluation2012
- KinectFusion2011
- A taxonomy and evaluation of dense two-frame stereo correspondence algorithms2002
- A multiple-baseline stereoIEEE Transactions on Pattern Analysis and Machine Intelligence, 1993
- Cooperative Computation of Stereo DisparityScience, 1976