Referring Multi-Object Tracking
Preprint
Abstract
Existing referring understanding tasks tend to involve the detection of a single text-referred object. In this paper, we propose a new and general referring understanding task, termed referring multi-object tracking (RMOT). Its core idea is to employ a language expression as a semantic cue to guide the prediction of multi-object tracking. To the best of our knowledge, it is the first work to achieve an arbitrary number of referent object predictions in videos. To push forward RMOT, we construct one benchmark with scalable expressions based on KITTI, named Refer-KITTI. Specifically, it provides 18 videos with 818 expressions, and each expression in a video is annotated with an average of 10.7 objects. Further, we develop a transformer-based architecture TransRMOT to tackle the new task in an online manner, which achieves impressive detection performance and outperforms other counterparts.Keywords
All Related Versions
This publication has 48 references indexed in Scilit:
- MOTR: End-to-End Multiple-Object Tracking with Transformer2022
- ByteTrack: Multi-object Tracking by Associating Every Detection Box2022
- FairMOT: On the Fairness of Detection and Re-identification in Multiple Object TrackingInternational Journal of Computer Vision, 2021
- URVOS: Unified Referring Video Object Segmentation Network with a Large-Scale Benchmark2020
- End-to-End Object Detection with Transformers2020
- HOTA: A Higher Order Metric for Evaluating Multi-object TrackingInternational Journal of Computer Vision, 2020
- Cosine Meets Softmax: A Tough-to-beat Baseline for Visual Grounding2020
- Video Object Segmentation with Language Referring Expressions2019
- Modeling Context in Referring Expressions2016
- Microsoft COCO: Common Objects in Context2014