Language-Guided Dual-Modal Local Correspondence for Single Object Tracking
- 29 August 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 26 (15209210) , 10637-10650
Abstract
This paper focuses on the advancement of single object tracking technologies in computer vision, which have broad applications including robotic vision, video surveillance, and sports video analysis. Current methods relying solely on the target's initial visual information encounter performance bottlenecks and limited applications, due to the scarcity of target semantics in appearance features and the continuous change in the target's appearance. To address these issues, we propose a novel approach, combining visual-language dual-modal single object tracking, that leverages natural language descriptions to enrich the semantic information of the moving target. We introduce a dual-modal single-object tracking algorithm based on local correspondence modeling. The algorithm decomposes visual features into multiple local visual semantic features and pairs them with local language features extracted from natural language descriptions. In addition, we also propose a new global relocalization method that utilizes visual language bimodal information to perceive target disappearance and misalignment and adaptively reposition the target in the entire image. This improves the tracker's ability to adapt to changes in target appearance over long periods of time, enabling long-term single target tracking based on bimodal semantic and motion information. Experimental results show that our model outperforms stateof-the-art methods, which demonstrates the effectiveness and efficiency of our approachKeywords
Funding Information
- National Natural Science Foundation of China (62276242, 62372153)
- National Aviation Science Foundation (2022Z071078001)
- CAAI-Huawei MindSpore Open Fund (CAAIXSJLJJ-2022-001A)
- Anhui Province Key Research and Development Program (202104a05020007)
- Dreams Foundation of Jianghuai Advance Technology Center (2023-ZM01Z001)
- USTC-IAT Application Sci. & Tech. Achievement Cultivation Program (JL06521001Y)
This publication has 53 references indexed in Scilit:
- Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework2022
- Visual object tracking: A surveyComputer Vision and Image Understanding, 2022
- Identifying content unaware features influencing popularity of videos on YouTube: A study based on seven regionsExpert Systems with Applications, 2022
- Ocean: Object-Aware Anchor-Free Tracking2020
- A survey on online learning for visual trackingThe Visual Computer, 2020
- Distractor-Aware Siamese Networks for Visual Object Tracking2018
- Fully-Convolutional Siamese Networks for Object Tracking2016
- Modeling Context in Referring Expressions2016
- Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking2016
- Exploiting the Circulant Structure of Tracking-by-Detection with Kernels2012