ToF and Stereo Data Fusion Using Dynamic Search Range Stereo Matching
- 7 June 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 24 (15209210) , 2739-2751
Abstract
Time-of-Flight (ToF) sensors and stereo vision systems are both widely used for capturing depth data. They have some complementary strengths and limitations, which have been exploited in prior research to produce more accurate depth maps by fusing data from the two sources. However, among these diverse data fusion approaches, none of them provides an end-to-end neural network solution. In this work, we propose the first end-to-end ToF and stereo data fusion network using the coarse-to-fine matching framework, where the prior of ToF depth is integrated into the stereo matching process by constraining the search range of stereo matching within an interval around the ToF camera depth measurement. We adopt a dynamic search range for each pixel according to an estimated ToF error map, which is more efficient and effective than a constant one when handling various errors. The ToF error map is estimated by the ToF error estimator branching out from the stereo matching network. Both ToF error estimation and stereo matching are performed in a joint framework, with the two tasks assisting each other mutually. We also propose an upsampling module to replace the naive bilinear upsampling in the coarse-to-fine stereo matching network, which reduces the error caused by the upsampling. The proposed deep network is trained end-to-end on synthetic datasets and generalizable to real-world datasets without further fine-tuning. Experimental results show that our fusion method achieves higher accuracy than either ToF or stereo alone, and outperforms state-of-the-art fusion methods on both synthetic and real data.Keywords
Funding Information
- National Key Research and Development Program of China (2018YFB1004904)
- Science and Technology Program of Suzhou City (SYG201920)
- National Natural Science Foundation of China (61972323)
- Key Program Special Fund in XJTLU (KSF-T-02, KSF-P-02)
This publication has 45 references indexed in Scilit:
- Probabilistic ToF and Stereo Data Fusion Based on Mixed Pixels Measurement ModelsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Fusion of Range and Stereo Data for High-Resolution Scene-ModelingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- A Survey on Time-of-Flight Stereo Fusion2013
- Locally Consistent ToF and Stereo Data Fusion2012
- High Accuracy TOF and Stereo Sensor Fusion at Interactive Rates2012
- Combining Stereo and Time-of-Flight Images with Application to Automatic Plant Phenotyping2011
- Depth Imaging by Combining Time-of-Flight and On-Demand StereoLecture Notes in Computer Science, 2009
- Fusion of stereo vision and Time-Of-Flight imaging for improved 3D estimationInternational Journal of Intelligent Systems Technologies and Applications, 2008
- A Combined Approach for Estimating Patchlets from PMD Depth Images and Stereo Intensity Images2007
- Hierarchical Warp Stereo1987