Abstract
This paper proposes a three-stream self-attention network (TSNet) for indoor semantic segmentation comprising two asymmetric input streams (asymmetric encoder structure) and a cross-modal distillation stream with a self-attention module. The two asymmetric input streams are ResNet34 for the red-green-blue (RGB) stream and VGGNet16 for the depth stream. Accompanying the RGB and depth streams, a cross-modal distillation stream with a self-attention module extracts new RGB plus depth features in each level in the bottom-up path. In addition, while using bilinear upsampling to recover the spatial resolution of the feature map, we incorporated the feature information of both the RGB flow and the depth flow through the self-attention module. We constructed two public datasets to evaluate the TSNet and achieved results comparable to those of current state-of-the-art methods.
Funding Information
  • National Natural Science Foundation of China (61502429, 61672337, 61972357)
  • Zhejiang Key R&D Program (2019C03135)
  • Natural Science Foundation of Zhejiang Province (LY18F020012)
  • China Postdoctoral Science Foundation (2015M581932)