SED-MVS: Segmentation-Driven and Edge-Aligned Deformation Multi-View Stereo With Depth Restoration and Occlusion Constraint
- 28 May 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 35 (11) , 11244-11257
Abstract
Recently, patch-deformation methods have exhibited significant effectiveness in multi-view stereo owing to the deformable and expandable patches in reconstructing textureless areas. However, existing approaches neglect to address the problem of deformation instability caused by easily overlooked edge-skipping, potentially leading to matching distortions, thus leaving room for further improvement. To fill this gap, we propose SED-MVS, which adopts panoptic segmentation and multi-trajectory diffusion strategy for segmentation-driven and edge-aligned patch deformation. Specifically, to prevent unanticipated edge-skipping, we first employ SAM2 for panoptic segmentation as depth-edge guidance to guide patch deformation, followed by multi-trajectory diffusion strategy to ensure patches are comprehensively aligned with depth edges. Moreover, to avoid potential inaccuracy of random initialization, we combine both sparse points from LoFTR and monocular depth map from DepthAnything V2 to restore reliable and realistic depth map for initialization and supervised guidance. Finally, we integrate the segmentation image with the monocular depth map to exploit inter-instance occlusion relationship, then further regard them as occlusion map to implement two distinct edge constraint, thereby facilitating occlusion-aware patch deformation. Extensive results on ETH3D, Tanks & Temples, BlendedMVS, Strecha and DL3DV-10K datasets validate the state-of-the-art performance and robust generalization capability of our proposed method.Keywords
Funding Information
- Strategic Priority Research Program of the Chinese Academy of Sciences (XDA0450203)
- National Natural Science Foundation of China (62172392)
- Beijing Smart Agriculture Innovation Consortium Project (BAlC10-2024)
- Basic Research Center, Innovation Program of Chinese Academy of Agricultural Sciences (CAAS-BRC-SAE-2025-01)
- Basic Research Center, Innovation Program of Chinese Academy of Agricultural Sciences (CAAS-ASTIP-2025-AII)
This publication has 44 references indexed in Scilit:
- TSAR-MVS: Textureless-aware segmentation and correlative refinement guided multi-view stereoPattern Recognition, 2024
- MVSTER: Epipolar Transformer for Efficient Multi-view Stereo2022
- BlendedMVS: A Large-Scale Dataset for Generalized Multi-View Stereo Networks2020
- Plane Completion and Filtering for Multi-View Stereo Reconstruction2019
- Multi-Scale Geometric Consistency Guided Multi-View Stereo2019
- Recurrent MVSNet for High-Resolution Multi-View Stereo Depth Inference2019
- MVSNet: Depth Inference for Unstructured Multi-view Stereo2018
- A Multi-view Stereo Benchmark with High-Resolution Images and Multi-camera Videos2017
- Pixelwise View Selection for Unstructured Multi-View Stereo2016
- On benchmarking camera calibration and multi-view stereo for high resolution imagery2008