S2Net: A Multitask Learning Network for Semantic Stereo of Satellite Image Pairs
- 28 November 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 62 (01962892) , 1-13
Abstract
Stereo matching and semantic segmentation are two significant tasks in remote sensing. Recently, deep learning approaches have been applied to these tasks separately. However, the lack of semantic supervision makes the training of stereo matching model susceptible to data disturbance, resulting in inferior generalization ability; foreground objects are sometimes confused with background pixels in RGB images, limiting the classification accuracy. By exploring the relationship between these two tasks, semantic stereo solves these problems simultaneously with multitask learning. Previous methods took semantic stereo as two parallel processing tasks, so they did not take full advantages of the additional information from both tasks and only obtained slight improvement. In this work, we designed a multitask learning framework semantic stereo network (S2Net). The proposed network generates cost volumes with feature maps supervised by semantic information to estimate disparity maps and fuses RGB-D feature maps to predict classification maps, therefore gathering multitask learning information. To enhance the performance of trained model, we also considered the continuity of disparity values and the duality of stereo image pairs in data augmentation. When applied in datasets without training, S2Net obtained 2.937% D1-Error in the WHU dataset, lower than 4.297% of the previous best method, depicting the generalization ability improvement from semantic supervision. In terms of semantic segmentation, the introduction of disparity maps increases the mean intersection over union (mIoU) from 61.375% to 69.096% in the US3D datasets. The experiments on the KITTI semantics benchmark show that our proposed method obtains 60.76% mIoU, achieving state-of-the-art among multitask learning methods.Keywords
Funding Information
- National Natural Science Foundation of China (42101346)
- China Postdoctoral Science Foundation (2020M680109)
This publication has 36 references indexed in Scilit:
- Efficient ConvNet for real-time semantic segmentation2017
- DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Dense Semantic Labeling of Subdecimeter Resolution Images With Convolutional Neural NetworksIEEE Transactions on Geoscience and Remote Sensing, 2016
- JOINT 3D ESTIMATION OF VEHICLES AND SCENE FLOWISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2015
- Displets: Resolving stereo ambiguities using object knowledge2015
- Fully convolutional networks for semantic segmentation2015
- Spatial Pyramid Pooling in Deep Convolutional Networks for Visual RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Stereo Processing by Semiglobal Matching and Mutual InformationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2007
- A Tutorial on the Cross-Entropy MethodAnnals of Operations Research, 2005
- THE DISTRIBUTION OF THE FLORA IN THE ALPINE ZONE.1New Phytologist, 1912