Exploring Pruning-Based Efficient Object Tracking via Hybrid Knowledge Distillation
- 12 September 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 36 (2) , 2433-2448
Abstract
Transformer-based trackers have demonstrated remarkable advancements in real-time tracking tasks on edge devices. Since lightweight backbone networks are typically designed for general-purpose tasks, our analysis reveals that, when applied to target tracking, they often contain structurally redundant layers, which limits the model’s efficiency. To address this issue, we propose a novel tracking framework that integrates backbone pruning with Hybrid Knowledge Distillation (HKD), effectively reducing model parameters and FLOPs while preserving high tracking accuracy. Inspired by the success of MiniLM and Focal and Global Distillation (FGD), we design a HKD framework tailored for tracking tasks. Our HKD introduces a multi-level and complementary distillation scheme, consisting of Token Distillation, Local Distillation, and Global Distillation. In Token Distillation, unlike MiniLM, which distills attention via QK dot-products and V, we disentangle and separately distill Q, K, and V representations to enhance structural attention alignment for tracking. For Local Distillation, we use the FGD concept by incorporating spatial foreground-background masks to capture region-specific discriminative cues more effectively. In Global Distillation, we use Vision Mamba module to model long-range dependencies and enhance semantic-level feature alignment. Our tracker HKDT achieves state-of-the-art (SOTA) performance across multiple datasets. On the GOT-10k benchmark, it demonstrates a groundbreaking 67.6% Average Overlap (AO), outperforming the current SOTA real-time tracker HiT-Base by 3.6% in accuracy while reducing computational costs by 64% and achieving 115% faster tracking speed on CPU platforms. The code and model will be available soon.Keywords
Funding Information
- National Natural Science Foundation of China (62476227)
This publication has 59 references indexed in Scilit:
- Exploring the Feature Extraction and Relation Modeling For Light-Weight Transformer Tracking2024
- Efficient Visual Tracking via Hierarchical Cross-Attention Transformer2023
- Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework2022
- FEAR: Fast, Efficient, Accurate and Robust Visual Tracker2022
- Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking2022
- AiATrack: Attention in Attention for Transformer Visual Tracking2022
- LaSOT: A High-quality Large-scale Single Object Tracking BenchmarkInternational Journal of Computer Vision, 2020
- TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild2018
- A Benchmark and Simulator for UAV Tracking2016
- Microsoft COCO: Common Objects in Context2014