MDFlow: Unsupervised Optical Flow Learning by Reliable Mutual Knowledge Distillation
- 8 September 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 33 (2) , 677-688
Abstract
Recent works have shown that optical flow can be learned by deep networks from unlabelled image pairs based on brightness constancy assumption and smoothness prior. Current approaches additionally impose an augmentation regularization term for continual self-supervision, which has been proved to be effective on difficult matching regions. However, this method also amplify the inevitable mismatch in unsupervised setting, blocking the learning process towards optimal solution. To break the dilemma, we propose a novel mutual distillation framework to transfer reliable knowledge back and forth between the teacher and student networks for alternate improvement. Concretely, taking estimation of off-the-shelf unsupervised approach as pseudo labels, our insight locates at defining a confidence selection mechanism to extract relative good matches, and then add diverse data augmentation for distilling adequate and reliable knowledge from teacher to student. Thanks to the decouple nature of our method, we can choose a stronger student architecture for sufficient learning. Finally, better student prediction is adopted to transfer knowledge back to the efficient teacher without additional costs in real deployment. Rather than formulating it as a supervised task, we find that introducing an extra unsupervised term for multi-target learning achieves best final results. Extensive experiments show that our approach, termed MDFlow, achieves state-of-the-art real-time accuracy and generalization ability on challenging benchmarks. Code is available at https://github.com/ltkong218/MDFlow.Keywords
All Related Versions
Funding Information
- NSFC, China (61876107, U1803261)
This publication has 62 references indexed in Scilit:
- Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness2016
- Video Stabilization for Strict Real-Time ApplicationsIEEE Transactions on Circuits and Systems for Video Technology, 2016
- Efficient Coarse-to-Fine Patch Match for Large Displacement Optical Flow2016
- A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation2016
- FlowNet: Learning Optical Flow with Convolutional Networks2015
- Object scene flow for autonomous vehicles2015
- A Naturalistic Open Source Movie for Optical Flow Evaluation2012
- Dense Point Trajectories by GPU-Accelerated Large Displacement Optical Flow2010
- Non-parametric local transforms for computing visual correspondence1994
- Determining optical flowArtificial Intelligence, 1981