SemStereo: Semantic-Constrained Stereo Matching Network for Remote Sensing
Open Access
- 11 April 2025
- journal article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (15) , 15758-15766
Abstract
Semantic segmentation and 3D reconstruction are two fundamental tasks in remote sensing, typically treated as separate or loosely coupled tasks. Despite attempts to integrate them into a unified network, the constraints between the two heterogeneous tasks are not explicitly modeled, since the pioneering studies either utilize a loosely coupled parallel structure or engage in only implicit interactions, failing to capture the inherent connections. In this work, we explore the connections between the two tasks and propose a new network that imposes semantic constraints on the stereo matching task, both implicitly and explicitly. Implicitly, we transform the traditional parallel structure to a new cascade structure termed Semantic-Guided Cascade structure, where the deep features enriched with semantic information are utilized for the computation of initial disparity maps, enhancing semantic guidance. Explicitly, we propose a Semantic Selective Refinement (SSR) module and a Left-Right Semantic Consistency (LRSC) module. The SSR refines the initial disparity map under the guidance of the semantic map. The LRSC ensures semantic consistency between two views via reducing the semantic divergence after transforming the semantic map from one view to the other using the disparity map. Experiments on the US3D and WHU datasets demonstrate that our method achieves state-of-the-art performance for both semantic segmentation and stereo matching.Keywords
This publication has 12 references indexed in Scilit:
- S3Net: Innovating Stereo Matching and Semantic Segmentation with a Single-Branch Semantic Stereo Network in Satellite Epipolar Imagery2024
- GABLE: A first fine-grained 3D building model of China on a national scale from very high resolution satellite imageryRemote Sensing of Environment, 2024
- HMSM-Net: Hierarchical multi-scale matching network for disparity estimation of high-resolution satellite stereo imagesISPRS Journal of Photogrammetry and Remote Sensing, 2022
- DisOptNet: Distilling Semantic Knowledge From Optical Images for Weather-Independent Building SegmentationIEEE Transactions on Geoscience and Remote Sensing, 2022
- PiCoCo: Pixelwise Contrast and Consistency Learning for Semisupervised Building Footprint SegmentationIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021
- PSRN: Polarimetric Space Reconstruction Network for PolSAR Image Semantic SegmentationIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021
- Large-Scale Semantic 3-D Reconstruction: Outcome of the 2019 IEEE GRSS Data Fusion Contest—Part AIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020
- EdgeStereo: An Effective Multi-task Learning Network for Stereo Matching and Edge DetectionInternational Journal of Computer Vision, 2020
- DispSegNet: Leveraging Semantics for End-to-End Learning of Disparity Estimation From Stereo ImageryIEEE Robotics and Automation Letters, 2019
- Advances in remote sensing applications for urban sustainabilityEuro-Mediterranean Journal for Environmental Integration, 2016