Joint Task-Recursive Learning for RGB-D Scene Understanding
- 10 July 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 42 (10) , 2608-2623
Abstract
RGB-D scene understanding under monocular camera is an emerging and challenging topic with many potential applications. In this paper, we propose a novel Task-Recursive Learning (TRL) framework to jointly and recurrently conduct three representative tasks therein containing depth estimation, surface normal prediction and semantic segmentation. TRL recursively refines the prediction results through a series of task-level interactions, where one-time cross-task interaction is abstracted as one network block of one time stage. In each stage, we serialize multiple tasks into a sequence and then recursively perform their interactions. To adaptively enhance counterpart patterns, we encapsulate interactions into a specific Task-Attentional Module (TAM) to mutually-boost the tasks from each other. Across stages, the historical experiences of previous states of tasks are selectively propagated into the next stages by using Feature-Selection unit (FS-Unit), which takes advantage of complementary information across tasks. The sequence of task-level interactions is also evolved along a coarse-to-fine scale space such that the required details may be refined progressively. Finally the task-abstracted sequence problem of multi-task prediction is framed into a recursive network. Extensive experiments on NYU-Depth v2 and SUN RGB-D datasets demonstrate that our method can recursively refines the results of the triple tasks and achieves state-of-the-art performance.Keywords
Funding Information
- National Natural Science Foundation of China (U1713208, 61772276, 61602244)
- Tencent AI Lab Rhino-Bird Focused Research Program (JR201922)
- Fundamental Research Funds for the Central Universities (30918011321)
- Guangdong Key Area Research (2018B010108003)
- Changjiang scholars
This publication has 53 references indexed in Scilit:
- Learning to segment with image-level annotationsPattern Recognition, 2016
- Learning Depth from Single Monocular Images Using Deep Convolutional Neural FieldsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Towards unified depth and semantic prediction from a single image2015
- Designing deep networks for surface normal estimation2015
- SUN RGB-D: A RGB-D scene understanding benchmark suite2015
- CNN Features Off-the-Shelf: An Astounding Baseline for Recognition2014
- The problem state: A cognitive bottleneck in multitasking.Journal of Experimental Psychology: Learning, Memory, and Cognition, 2010
- Learning long‐range vision for autonomous off‐road drivingJournal of Field Robotics, 2009
- High speed obstacle avoidance using monocular vision and reinforcement learning2005
- Regularized multi--task learning2004