MAST: A Memory-Augmented Self-Supervised Tracker
Authors:
- 1 June 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 6478-6487
Abstract
Recent interest in self-supervised dense tracking has yielded rapid progress, but performance still remains far from supervised methods. We propose a dense tracking model trained on videos without any annotations that surpasses previous self-supervised methods on existing benchmarks by a significant margin (+15%), and achieves performance comparable to supervised methods. In this paper, we first reassess the traditional choices used for self-supervised training and reconstruction loss by conducting thorough experiments that finally elucidate the optimal choices. Second, we further improve on existing methods by augmenting our architecture with a crucial memory component. Third, we benchmark on large-scale semi-supervised video object segmentation~(aka. dense tracking), and propose a new metric: generalizability. Our first two contributions yield a self-supervised network that for the first time is competitive with supervised methods on standard evaluation metrics of dense tracking. When measuring generalizability, we show self-supervised approaches are actually superior to the majority of supervised methods. We believe this new generalizability metric can better capture the real-world use-cases for dense tracking, and will spur new interest in this research direction.Keywords
All Related Versions
This publication has 35 references indexed in Scilit:
- Slow and Steady Feature Analysis: Higher Order Temporal Coherence in Video2016
- A Novel Performance Evaluation Methodology for Single-Target TrackersIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Learning to See by Moving2015
- Learning Image Representations Tied to Ego-Motion2015
- Unsupervised Learning of Visual Representations Using Videos2015
- Describing Videos by Exploiting Temporal Structure2015
- Color transfer between imagesIEEE Computer Graphics and Applications, 2001
- Long Short-Term MemoryNeural Computation, 1997
- Eye–hand coordination in the newborn.Developmental Psychology, 1982
- Smooth-Pursuit Eye Movements in the Newborn InfantChild Development, 1979