An Adaptive Depth Processing System Based on Foundation Transformers and Time-of-Flight Fusion
- 15 December 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems I: Regular Papers
- Vol. 73 (4) , 2647-2658
Abstract
Generating high-quality depth maps with accurate values is a critical research topic, and various depth estimation methods, such as Time-of-Flight (ToF) and Monocular Depth Estimation (MDE), are advancing rapidly. However, each single-sensor approach is inherently limited by the characteristics of its respective modality. Meanwhile, fused approaches using multiple sensors or dedicated trained models are often plagued by system complexity and limited generalization. In this paper, we propose an adaptive depth processing system based on the foundation transformer and ToF fusion, aiming to harness the precision of ToF data and the high-quality depth priors provided by the foundation model in a monocular and zero-shot manner. A hardware–algorithm co-design partitions the computation between a dedicated ASIC for compute-intensive foundation transformer acceleration and an MPSoC FPGA for potential reconfigurable fusion, yielding 36.4 ms latency and 1.9 W power under a 27.6 GOPS/frame workload. Zero-shot evaluations on various ToF data sources including FLAT, TICaM, and in-house captures confirm strong cross-domain generalization, demonstrating the system’s adaptability, efficiency, and accuracy.Keywords
Funding Information
- ACCESS-AI Chip Center for Emerging Smart Systems, sponsored by the InnoHK initiative of the Innovation and Technology Commission of the Hong Kong Special Administrative Region Government.
This publication has 39 references indexed in Scilit:
- An Efficient Unsupervised Learning-based Monocular Depth Estimation Processor with Partial-Switchable Systolic Array Architecture in Edge Devices2022
- Learning the Depths of Moving People by Watching Frozen People2019
- DDRNet: Depth Map Denoising and Refinement for Consumer Depth Cameras Using Cascaded CNNs2018
- Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset2018
- Deep End-to-End Time-of-Flight Imaging2018
- Deep Ordinal Regression Network for Monocular Depth Estimation2018
- Efficient Multi-frequency Phase Unwrapping Using Kernel Density Estimation2016
- Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue2016
- Monocular Depth Estimation Using Neural Regression Forest2016
- Deep convolutional neural fields for depth estimation from a single image2015