UniSOT: A Unified Framework for Multi-Modality Single Object Tracking
Authors:
- Yinchao MaYMYinchao Ma
- Yuyang TangYTYuyang Tang
- Wenfei YangWYWenfei Yang
- Tianzhu ZhangTZTianzhu Zhang
- Xu ZhouXZXu Zhou
- Feng WuFWFeng Wu
Preprint
Abstract
Single object tracking aims to localize target object with specific reference modalities (bounding box, natural language or both) in a sequence of specific video modalities (RGB, RGB+Depth, RGB+Thermal or RGB+Event.). Different reference modalities enable various human-machine interactions, and different video modalities are demanded in complex scenarios to enhance tracking robustness. Existing trackers are designed for single or several video modalities with single or several reference modalities, which leads to separate model designs and limits practical applications. Practically, a unified tracker is needed to handle various requirements. To the best of our knowledge, there is still no tracker that can perform tracking with these above reference modalities across these video modalities simultaneously. Thus, in this paper, we present a unified tracker, UniSOT, for different combinations of three reference modalities and four video modalities with uniform parameters. Extensive experimental results on 18 visual tracking, vision-language tracking and RGB+X tracking benchmarks demonstrate that UniSOT shows superior performance against modality-specific counterparts. Notably, UniSOT outperforms previous counterparts by over 3.0\% AUC on TNL2K across all three reference modalities and outperforms Un-Track by over 2.0\% main metric across all three RGB+X video modalities.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Middle fusion and multi-stage, multi-form prompts for robust RGB-T trackingNeurocomputing, 2024
- VisEvent: Reliable Object Tracking via Collaboration of Frame and Event FlowsIEEE Transactions on Cybernetics, 2023
- Motion-Driven Tracking via End-to-End Coarse-to-Fine VerifyingIEEE Transactions on Circuits and Systems for Video Technology, 2023
- Joint Classification and Regression for Visual Tracking with Fully Convolutional Siamese NetworksInternational Journal of Computer Vision, 2022
- The Ninth Visual Object Tracking VOT2021 Challenge Results2021
- Adaptive Channel Selection for Robust Visual Object Tracking with Discriminative Correlation FiltersInternational Journal of Computer Vision, 2021
- LaSOT: A High-quality Large-scale Single Object Tracking BenchmarkInternational Journal of Computer Vision, 2020
- Cross-Modal Pattern-Propagation for RGB-T Tracking2020
- RGB-T object tracking: Benchmark and baselinePattern Recognition, 2019
- Robust Visual Tracking via Structured Multi-Task Sparse LearningInternational Journal of Computer Vision, 2012