TSNet: Three-Stream Self-Attention Network for RGB-D Indoor Semantic Segmentation
- 10 June 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Intelligent Systems
- Vol. 36 (4) , 73-78
Abstract
This paper proposes a three-stream self-attention network (TSNet) for indoor semantic segmentation comprising two asymmetric input streams (asymmetric encoder structure) and a cross-modal distillation stream with a self-attention module. The two asymmetric input streams are ResNet34 for the red-green-blue (RGB) stream and VGGNet16 for the depth stream. Accompanying the RGB and depth streams, a cross-modal distillation stream with a self-attention module extracts new RGB plus depth features in each level in the bottom-up path. In addition, while using bilinear upsampling to recover the spatial resolution of the feature map, we incorporated the feature information of both the RGB flow and the depth flow through the self-attention module. We constructed two public datasets to evaluate the TSNet and achieved results comparable to those of current state-of-the-art methods.Keywords
Funding Information
- National Natural Science Foundation of China (61502429, 61672337, 61972357)
- Zhejiang Key R&D Program (2019C03135)
- Natural Science Foundation of Zhejiang Province (LY18F020012)
- China Postdoctoral Science Foundation (2015M581932)
This publication has 7 references indexed in Scilit:
- Joint Task-Recursive Learning for RGB-D Scene UnderstandingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2019
- Three-Stream Attention-Aware Network for RGB-D Salient Object DetectionIEEE Transactions on Image Processing, 2019
- RGB-D joint modelling with scene geometric information for indoor semantic segmentationMultimedia Tools and Applications, 2018
- A fusion network for semantic segmentation using RGB-D data2018
- STD2P: RGBD Semantic Segmentation Using Spatio-Temporal Data-Driven Pooling2017
- Fully convolutional networks for semantic segmentation2015
- Learning Rich Features from RGB-D Images for Object Detection and Segmentation2014