Monocular Depth and Ego-motion Estimation with Scale Based on Superpixel and Normal Constraints
Open Access
- 30 October 2024
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 20 (10) , 1-26
Abstract
Three-dimensional perception in intelligent virtual and augmented reality (VR/AR) and autonomous vehicles (AV) applications is critical and attracting significant attention. The self-supervised monocular depth and ego-motion estimation serves as a more intelligent learning approach that provides the required scene depth and location for 3D perception. However, the existing self-supervised learning methods suffer from scale ambiguity, boundary blur, and imbalanced depth distribution, limiting the practical applications of VR/AR and AV. In this article, we propose a new self-supervised learning framework based on superpixel and normal constraints to address these problems. Specifically, we formulate a novel 3D edge structure consistency loss to alleviate the boundary blur of depth estimation. To address the scale ambiguity of estimated depth and ego-motion, we propose a novel surface normal network for efficient camera height estimation. The surface normal network is composed of a deep fusion module and a full-scale hierarchical feature aggregation module. Meanwhile, to realize the global smoothing and boundary discriminability of the predicted normal map, we introduce a novel fusion loss which is based on the consistency constraints of the normal in edge domains and superpixel regions. Experiments are conducted on several benchmarks, and the results illustrate that the proposed approach outperforms the state-of-the-art methods in depth, ego-motion, and surface normal estimation.Keywords
Funding Information
- National Natural Science Foundation of China (U2033218)
- Shanghai Local Capacity Enhancement project (21010501500)
- “Science and Technology Innovation Action Plan” of Shanghai Science and Technology Commission for Social Development Project (21DZ1204900)
This publication has 77 references indexed in Scilit:
- Self-Supervised Learning of Depth and Ego-Motion for 3D Perception in Human Computer InteractionACM Transactions on Multimedia Computing, Communications, and Applications, 2023
- MLDA-Net: Multi-Level Dual Attention-Based Network for Self-Supervised Monocular Depth EstimationIEEE Transactions on Image Processing, 2021
- Adversarial Learning for Joint Optimization of Depth and Ego-MotionIEEE Transactions on Image Processing, 2020
- DVL-SLAM: sparse depth enhanced direct visual-LiDAR SLAMAutonomous Robots, 2019
- Ground-Plane-Based Absolute Scale Estimation for Monocular Visual OdometryIEEE Transactions on Intelligent Transportation Systems, 2019
- DF-Net: Unsupervised Joint Learning of Depth and Flow Using Cross-Task Consistency2018
- Designing and Evaluating a Mesh Simplification Algorithm for Virtual RealityACM Transactions on Multimedia Computing, Communications, and Applications, 2018
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Vision meets robotics: The KITTI datasetThe International Journal of Robotics Research, 2013
- Virtual and augmented reality 2020IEEE Computer Graphics and Applications, 2000