Beyond the Field-of-View: Enhancing Scene Visibility and Perception with Clip-Recurrent Transformer
Authors:
- 24 June 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. PP (23798858) , 1-16
Abstract
Vision sensors are widely applied in vehicles, robots, and roadside infrastructure. However, due to limitations in hardware cost and system size, camera Field-of-View (FoV) is often restricted and may not provide sufficient coverage. Nevertheless, from a spatiotemporal perspective, it is possible to obtain information beyond the camera's physical FoV from past video streams. In this paper, we propose the concept of online video inpainting for autonomous vehicles to expand the field of view, thereby enhancing scene visibility, perception, and system safety. To achieve this, we introduce the FlowLens architecture, which explicitly employs optical flow and implicitly incorporates a novel clip-recurrent transformer for feature propagation. FlowLens offers two key features: 1) FlowLens includes a newly designed Clip-Recurrent Hub with 3D-Decoupled Cross Attention (DDCA) to progressively process global information accumulated over time. 2) It integrates a multi-branch Mix Fusion Feed Forward Network (MixF3N) to enhance the precise spatial flow of local features. To facilitate training and evaluation, we derive the KITTI360 dataset with various FoV mask, which covers both outer- and inner FoV expansion scenarios. We also conduct quantitative assessments of beyond-FoV semantics across different models and perform qualitative comparisons of beyond-FoV object detection. We illustrate that employing FlowLens to reconstruct unseen scenes even enhances perception within the field of view by providing reliable semantic context. Extensive experiments and user studies involving offline and online video inpainting, as well as beyond-FoV perception tasks, demonstrate that FlowLens achieves state-of-the-art performance. The source code and dataset are made publicly available at https://github.com/MasterHow/FlowLens .Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (12174341)
This publication has 75 references indexed in Scilit:
- Deep 360$^\circ $ Optical Flow Estimation Based on Multi-projection Fusion2022
- Flow-Guided Transformer for Video Inpainting2022
- Error Compensation Framework for Flow-Guided Video Inpainting2022
- RAFT: Recurrent All-Pairs Field Transforms for Optical Flow2020
- Learning Joint Spatial-Temporal Transformations for Video Inpainting2020
- Flow-edge Guided Video Completion2020
- Learning Blind Video Temporal Consistency2018
- Image Inpainting for Irregular Holes Using Partial Convolutions2018
- OmniDepth: Dense Depth Estimation for Indoors Spherical Panoramas2018
- YouTube-VOS: Sequence-to-Sequence Video Object Segmentation2018