MS-RAFT+: High Resolution Multi-Scale RAFT
Open Access
- 18 December 2023
- journal article
- research article
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 132 (5) , 1835-1856
Abstract
Hierarchical concepts have proven useful in many classical and learning-based optical flow methods regarding both accuracy and robustness. In this paper we show that such concepts are still useful in the context of recent neural networks that follow RAFT’s paradigm refraining from hierarchical strategies by relying on recurrent updates based on a single-scale all-pairs transform. To this end, we introduce MS-RAFT+: a novel recurrent multi-scale architecture based on RAFT that unifies several successful hierarchical concepts. It employs a coarse-to-fine estimation to enable the use of finer resolutions by useful initializations from coarser scales. Moreover, it relies on RAFT’s correlation pyramid that allows to consider non-local cost information during the matching process. Furthermore, it makes use of advanced multi-scale features that incorporate high-level information from coarser scales. And finally, our method is trained subject to a sample-wise robust multi-scale multi-iteration loss that closely supervises each iteration on each scale, while allowing to discard particularly difficult samples. In combination with an appropriate mixed-dataset training strategy, our method performs favorably. It not only yields highly accurate results on the four major benchmarks (KITTI 2015, MPI Sintel, Middlebury and VIPER), it also allows to achieve these results with a single model and a single parameter setting. Our trained model and code are available at https://github.com/cv-stuttgart/MS_RAFT_plus.Keywords
Funding Information
- Deutsche Forschungsgemeinschaft (Project-ID 251654672)
This publication has 76 references indexed in Scilit:
- Weighted local intensity fusion method for variational optical flow estimationPattern Recognition, 2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Visual object tracking—classical and contemporary approachesFrontiers of Computer Science, 2015
- A Naturalistic Open Source Movie for Optical Flow Evaluation2012
- A Database and Evaluation Methodology for Optical FlowInternational Journal of Computer Vision, 2010
- PatchMatchACM Transactions on Graphics, 2009
- High Accuracy Optical Flow Estimation Based on a Theory for Warping2004
- The use of optical flow for road navigationIEEE Transactions on Robotics and Automation, 1998
- A computational framework and an algorithm for the measurement of visual motionInternational Journal of Computer Vision, 1989
- Investigations of multigrid algorithms for the estimation of optical flow fields in image sequencesComputer Vision, Graphics, and Image Processing, 1988