Zhiyi Mo
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Zhiyi Mo has a total of 32 co-authors. Here are the top co-authors by number of shared publications:
- Qihua Liang
Guangxi Normal University
9 shared publications - Bineng Zhong
Guangxi Normal University
8 shared publications - Shuxiang Song
Guangxi Normal University
5 shared publications - Feng Shao
Ningbo University
4 shared publications - Ning Li
Guangxi Normal University
4 shared publications - Hangwei Chen
Ningbo University
3 shared publications
Latest Publications
- 1 November 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Consumer Electronics
- Vol. 72 (4) , 8070-8087
PU-T2M: Heterogeneous Knowledge Distilling Network From Transformer to Mamba for Lightweight Point Cloud Upsampling
Three-dimensional point clouds have become a core sensing modality in modern consumer electronics systems, particularly in intelligent vehicles equipped with low-power LiDAR and depth cameras. However, the point clouds captured by these consumer-grade sensors are often sparse and noisy due to complex environment and sensor constraints. Such degradation adversely affects downstream perception tasks, including real-time scene reconstruction, object understanding, and navigation stability. Point Cloud Upsampling (PCU) is therefore essential for restoring structural fidelity and enabling reliable three dimensions (3D) environment modeling in resource-constrained edge-to-cloud automotive systems. To overcome these issues, we propose a novel lightweight PCU architecture based on Heterogeneous Knowledge Distilling Network from Transformer to Mamba, termed PU-T2M. This workflow achieves state-of-the-art (SOTA) performance while being computationally efficient, making it suitable for edge devices with limited resources. Specifically, our network distills dark knowledge from Transformer to Mamba, enabling the model to learn long-range global context while preserving local geometric structure. We incorporate Mamba and knowledge distillation into the PCU task to achieve a more efficient and effective modeling framework. Specifically, we design a dual-alignment distillation loss that facilitates knowledge transfer while enforcing both explicit alignment in point distributions and implicit alignment in feature representations. In addition, we develop a Global Context Preserver module and a Local Geometry Preserver module to strengthen the Transformer’s global contextual modeling capability and enhance Mamba’s capacity for fine-grained local geometric learning. Extensive experiments conducted on four benchmark datasets demonstrate that PU-T2M attains state-of-the-art performance while maintaining a compact parameter scale and strong robustness. The proposed approach enables accurate and computationally efficient point cloud reconstruction, making it particularly suitable for industrial metrology and high-precision instrument-level quality inspection scenarios.
- 1 November 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Consumer Electronics
- Vol. 72 (4) , 8070-8087
PU-T2M: Heterogeneous Knowledge Distilling Network From Transformer to Mamba for Lightweight Point Cloud Upsampling
Three-dimensional point clouds have become a core sensing modality in modern consumer electronics systems, particularly in intelligent vehicles equipped with low-power LiDAR and depth cameras. However, the point clouds captured by these consumer-grade sensors are often sparse and noisy due to complex environment and sensor constraints. Such degradation adversely affects downstream perception tasks, including real-time scene reconstruction, object understanding, and navigation stability. Point Cloud Upsampling (PCU) is therefore essential for restoring structural fidelity and enabling reliable three dimensions (3D) environment modeling in resource-constrained edge-to-cloud automotive systems. To overcome these issues, we propose a novel lightweight PCU architecture based on Heterogeneous Knowledge Distilling Network from Transformer to Mamba, termed PU-T2M. This workflow achieves state-of-the-art (SOTA) performance while being computationally efficient, making it suitable for edge devices with limited resources. Specifically, our network distills dark knowledge from Transformer to Mamba, enabling the model to learn long-range global context while preserving local geometric structure. We incorporate Mamba and knowledge distillation into the PCU task to achieve a more efficient and effective modeling framework. Specifically, we design a dual-alignment distillation loss that facilitates knowledge transfer while enforcing both explicit alignment in point distributions and implicit alignment in feature representations. In addition, we develop a Global Context Preserver module and a Local Geometry Preserver module to strengthen the Transformer’s global contextual modeling capability and enhance Mamba’s capacity for fine-grained local geometric learning. Extensive experiments conducted on four benchmark datasets demonstrate that PU-T2M attains state-of-the-art performance while maintaining a compact parameter scale and strong robustness. The proposed approach enables accurate and computationally efficient point cloud reconstruction, making it particularly suitable for industrial metrology and high-precision instrument-level quality inspection scenarios.
- 10 June 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. PP (16) , 1
PU-TransMamba: A Hybrid Point Cloud Upsampling Framework with Detail-Aware Transformer and Spatially-Coherent Mamba
High-quality 3D point clouds are essential for high-fidelity perception in Internet of Things (IoT)-enabled intelligent systems. While Point Cloud Upsampling (PCU) is widely used to mitigate data sparsity, existing methods often struggle to balance the preservation of fine-grained local details with the maintenance of global topological consistency. Transformer-based approaches frequently suffer from excessive computational overhead and high-frequency detail loss, whereas emerging state space models like Mamba, despite their efficiency, inevitably sacrifice spatial coherence due to the 1D serialization of irregular 3D points. To address these critical bottlenecks, we introduce PU-TransMamba, a hybrid framework that synergistically leverages a Detail-Aware Transformer and a Spatially-Coherent Mamba. Each component is designed to resolve specific PCU limitations: a Complexity-Aware Bilateral Decoder is developed to adaptively recover sharp geometric edges by processing features across dual domains, while a Sequence-Aligned Mamba Encoder utilizes multiple spatial curvature descriptors to compensate for the spatial information loss inherent in serialization. Additionally, a Global Geometry Injector and a Local Neighbor Injector are designed to ensure structural integrity by infusing holistic skeletal priors and neighborhood context, respectively. To minimize feature discrepancies between the hybrid branches, we also propose a Self-Distillation Loss. Extensive experiments on five benchmark datasets demonstrate that PU-TransMamba outperforms state-of-the-art methods in both reconstruction accuracy and computational scalability. The results confirm its ability to recover intricate geometries, indicating significant potential for IoT-driven 3D perception and communication systems.
- 10 June 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. PP (16) , 1
PU-TransMamba: A Hybrid Point Cloud Upsampling Framework with Detail-Aware Transformer and Spatially-Coherent Mamba
High-quality 3D point clouds are essential for high-fidelity perception in Internet of Things (IoT)-enabled intelligent systems. While Point Cloud Upsampling (PCU) is widely used to mitigate data sparsity, existing methods often struggle to balance the preservation of fine-grained local details with the maintenance of global topological consistency. Transformer-based approaches frequently suffer from excessive computational overhead and high-frequency detail loss, whereas emerging state space models like Mamba, despite their efficiency, inevitably sacrifice spatial coherence due to the 1D serialization of irregular 3D points. To address these critical bottlenecks, we introduce PU-TransMamba, a hybrid framework that synergistically leverages a Detail-Aware Transformer and a Spatially-Coherent Mamba. Each component is designed to resolve specific PCU limitations: a Complexity-Aware Bilateral Decoder is developed to adaptively recover sharp geometric edges by processing features across dual domains, while a Sequence-Aligned Mamba Encoder utilizes multiple spatial curvature descriptors to compensate for the spatial information loss inherent in serialization. Additionally, a Global Geometry Injector and a Local Neighbor Injector are designed to ensure structural integrity by infusing holistic skeletal priors and neighborhood context, respectively. To minimize feature discrepancies between the hybrid branches, we also propose a Self-Distillation Loss. Extensive experiments on five benchmark datasets demonstrate that PU-TransMamba outperforms state-of-the-art methods in both reconstruction accuracy and computational scalability. The results confirm its ability to recover intricate geometries, indicating significant potential for IoT-driven 3D perception and communication systems.
- 12 March 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. PP (15209210) , 1-10
UBATrack: Spatio-Temporal State Space Model for General Multi-Modal Tracking
Multi-modal object tracking has attracted consider able attention by integrating multiple complementary inputs (e.g., thermal, depth, and event data) to achieve outstanding performance. Although current general-purpose multi-modal trackers primarily unify various modal tracking tasks (i.e., RGB-Thermal infrared, RGB-Depth or RGB-Event tracking) through prompt learning, they still overlook the effective capture of spatio-temporal cues. In this work, we introduce a novel multi-modal tracking framework based on a mamba-style state space model, termed UBATrack. Our UBATrack comprises two simple yet effective modules: a Spatio-temporal Mamba Adapter (STMA) and a Dynamic Multi-modal Feature Mixer. The former leverages Mamba's long-sequence modeling capability to jointly model cross-modal dependencies and spatio-temporal visual cues in an adapter-tuning manner. The latter further enhances multi-modal representation capacity across multiple feature dimensions to improve tracking robustness. In this way, UBATrack eliminates the need for costly full-parameter fine-tuning, thereby improving the training efficiency of multi-modal tracking algorithms. Experiments show that UBATrack outperforms state-of-the-art methods on RGB-T, RGB-D, and RGB-E tracking benchmarks, achieving outstanding results on the LasHeR, RGBT234, RGBT210, DepthTrack, VOT-RGBD22, and VisEvent datasets.
- 12 March 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. PP (15209210) , 1-10
UBATrack: Spatio-Temporal State Space Model for General Multi-Modal Tracking
Multi-modal object tracking has attracted consider able attention by integrating multiple complementary inputs (e.g., thermal, depth, and event data) to achieve outstanding performance. Although current general-purpose multi-modal trackers primarily unify various modal tracking tasks (i.e., RGB-Thermal infrared, RGB-Depth or RGB-Event tracking) through prompt learning, they still overlook the effective capture of spatio-temporal cues. In this work, we introduce a novel multi-modal tracking framework based on a mamba-style state space model, termed UBATrack. Our UBATrack comprises two simple yet effective modules: a Spatio-temporal Mamba Adapter (STMA) and a Dynamic Multi-modal Feature Mixer. The former leverages Mamba's long-sequence modeling capability to jointly model cross-modal dependencies and spatio-temporal visual cues in an adapter-tuning manner. The latter further enhances multi-modal representation capacity across multiple feature dimensions to improve tracking robustness. In this way, UBATrack eliminates the need for costly full-parameter fine-tuning, thereby improving the training efficiency of multi-modal tracking algorithms. Experiments show that UBATrack outperforms state-of-the-art methods on RGB-T, RGB-D, and RGB-E tracking benchmarks, achieving outstanding results on the LasHeR, RGBT234, RGBT210, DepthTrack, VOT-RGBD22, and VisEvent datasets.
- 27 February 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 13 (10) , 21556-21570
BigCounter: A Bidirectional-Guided Network With Scene-Semantics-Driven Fusion for RGB–Thermal Crowd Counting
Accurate crowd counting has become increasingly essential for public safety management and Internet of Video Things (IoVT) applications, driven by rapid population growth and urbanization. However, RGB and thermal (RGB–T) crowd counting remains challenging due to poor recognition of small targets and degraded performance under extreme conditions such as low-light environments. To address these issues, we propose a bidirectional-guided network with scene-semantics-driven fusion for RGB–T crowd counting (BigCounter) that enhances robustness and generalization in complex scenes. BigCounter comprises three parallel branches: a primary branch, a dynamic illumination auxiliary enhancement branch (DIAEB), and a high-resolution auxiliary enhancement branch (HAEB), which, respectively, improve robustness under illumination variations and accuracy for small-target detection. Moreover, a cross-layer scene-driven fusion module (CLSFM) and a cross-modal semantic-driven fusion module (CMSFM) are designed to strengthen structural consistency and explore semantic complementarity between modalities. Through multibranch collaboration and semantic-aware fusion, BigCounter significantly enhances feature representation. Extensive experiments on two benchmark RGB–T datasets demonstrate that BigCounter achieves superior accuracy and generalization compared with state-of-the-art (SOTA) methods.
- 27 February 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 13 (10) , 21556-21570
BigCounter: A Bidirectional-Guided Network With Scene-Semantics-Driven Fusion for RGB–Thermal Crowd Counting
Accurate crowd counting has become increasingly essential for public safety management and Internet of Video Things (IoVT) applications, driven by rapid population growth and urbanization. However, RGB and thermal (RGB–T) crowd counting remains challenging due to poor recognition of small targets and degraded performance under extreme conditions such as low-light environments. To address these issues, we propose a bidirectional-guided network with scene-semantics-driven fusion for RGB–T crowd counting (BigCounter) that enhances robustness and generalization in complex scenes. BigCounter comprises three parallel branches: a primary branch, a dynamic illumination auxiliary enhancement branch (DIAEB), and a high-resolution auxiliary enhancement branch (HAEB), which, respectively, improve robustness under illumination variations and accuracy for small-target detection. Moreover, a cross-layer scene-driven fusion module (CLSFM) and a cross-modal semantic-driven fusion module (CMSFM) are designed to strengthen structural consistency and explore semantic complementarity between modalities. Through multibranch collaboration and semantic-aware fusion, BigCounter significantly enhances feature representation. Extensive experiments on two benchmark RGB–T datasets demonstrate that BigCounter achieves superior accuracy and generalization compared with state-of-the-art (SOTA) methods.
- 13 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 63 (01962892) , 1-15
Adversarial Robust Salient Object Detection in Optical Remote Sensing Images With Implicit Feature Enhancement
Deep neural networks (DNNs) have achieved significant progress in optical remote sensing images salient object detection (ORSI-SOD) and are widely applied to various remote sensing image analysis tasks. However, few SOD models demonstrate robust performance under adversarial perturbations, which ultimately leads to a decline in detection accuracy. Moreover, most existing defense methods inject fixed Gaussian noise globally into the image. Although such approaches are easy to implement, they have several limitations in inaccurate uncertainty estimation and neglecting the unique characteristics of local salient regions. Furthermore, existing adversarial defense research rarely addresses the challenges specific to the ORSI-SOD task, leaving a gap in effective defense strategies. To tackle these issues, we propose a novel defense method, which enhances the adversarial robustness of ORSI-SOD models through implicit feature enhancement. The algorithm first proposes a two-stage strategy of reverse local noise search and forward global noise optimization, enhancing generalization ability by implicitly enhancing features to better simulate network uncertainty. Then, the algorithm proposes a global-guided texture information enhancement (GTIE) module for low-level features and a global-guided semantics information enhancement (GSIE) module for high-level features, focusing on strengthening low-level texture information and enhancing the model’s understanding of high-level contextual semantic features, respectively. This dual-module design effectively weakens the impact of adversarial noise, significantly improving the robustness and accuracy of object detection. Extensive experiments on three ORSI-SOD datasets demonstrate that our defense strategy better estimates the uncertainty, resulting in an average performance improvement of 23.2% in $F_{\beta } ^{\mathrm { max}}$ and 34.1% in $E_{\xi } ^{\mathrm { max}}$ across six ORSI-SOD models under five different adversarial attack methods. Our code will be released in the public repository at https://github.com/kexi0714/IFe.
- 13 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 63 (01962892) , 1-15
Adversarial Robust Salient Object Detection in Optical Remote Sensing Images With Implicit Feature Enhancement
Deep neural networks (DNNs) have achieved significant progress in optical remote sensing images salient object detection (ORSI-SOD) and are widely applied to various remote sensing image analysis tasks. However, few SOD models demonstrate robust performance under adversarial perturbations, which ultimately leads to a decline in detection accuracy. Moreover, most existing defense methods inject fixed Gaussian noise globally into the image. Although such approaches are easy to implement, they have several limitations in inaccurate uncertainty estimation and neglecting the unique characteristics of local salient regions. Furthermore, existing adversarial defense research rarely addresses the challenges specific to the ORSI-SOD task, leaving a gap in effective defense strategies. To tackle these issues, we propose a novel defense method, which enhances the adversarial robustness of ORSI-SOD models through implicit feature enhancement. The algorithm first proposes a two-stage strategy of reverse local noise search and forward global noise optimization, enhancing generalization ability by implicitly enhancing features to better simulate network uncertainty. Then, the algorithm proposes a global-guided texture information enhancement (GTIE) module for low-level features and a global-guided semantics information enhancement (GSIE) module for high-level features, focusing on strengthening low-level texture information and enhancing the model’s understanding of high-level contextual semantic features, respectively. This dual-module design effectively weakens the impact of adversarial noise, significantly improving the robustness and accuracy of object detection. Extensive experiments on three ORSI-SOD datasets demonstrate that our defense strategy better estimates the uncertainty, resulting in an average performance improvement of 23.2% in $F_{\beta } ^{\mathrm { max}}$ and 34.1% in $E_{\xi } ^{\mathrm { max}}$ across six ORSI-SOD models under five different adversarial attack methods. Our code will be released in the public repository at https://github.com/kexi0714/IFe.
- 20 June 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 6369-6381
Robust Multi-Stage Tracking via Multi-Scale and Multi-Level Representation Learning
How to learn multi-scale and multi-level representations is crucial for robust tracking. However, most current one-stream structure based trackers with visual transformers (dubbed ViTs) cannot effectively capture multi-scale representations due to the structure of their adopted ViTs is non-hierarchical. Meanwhile, they often only use the output features from the final layer for predicting results (i.e., ignoring the utilization of low-level features from the shallow layers) which may result in a certain degree of lacking multi-level representation learning ability. To address these issues, we propose a robust multi-stage tracker that effectively combines the advantages of both hierarchical and one-stream structured ViT as a tracking backbone to improve the multi-scale and multi-level representation learning abilities. Specifically, first of all, we design a hierarchical tracker with a three-stage backbone. In the first two stages of our tracker, we utilize a dual-branch structure to obtain multi-scale features of the template and search region separately. Especially, We design the local scale awareness modules based on simple MLP layers to capture multi-scale features. These modules remove complex operations such as convolutions or shifted window attentions, thus avoiding the performance degradation caused by traditional hierarchical ViTs. In the third stage (i.e. the main stage), we construct a global encoder based on the one-stream ViT to achieve efficient feature extraction and feature interaction for our tracker. Then, we design a multi-level feature integration module in the main stage to explicitly utilize the representation information learned from the shallow layers and fuse them with the features of the final layer to obtain multi-level representation information. Lastly, benefit from the these designs, our tracker can effectively capture more multi-scale and multi-level representations for robust tracking. Comprehensive experiments on GOT-10k, LaSOT, LaSOT$_{ext}$, TNL2K, UAV123, TrackingNet and VOT2020 benchmarks validate the effectiveness and robustness of our method.
- 20 June 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 6369-6381
Robust Multi-Stage Tracking via Multi-Scale and Multi-Level Representation Learning
How to learn multi-scale and multi-level representations is crucial for robust tracking. However, most current one-stream structure based trackers with visual transformers (dubbed ViTs) cannot effectively capture multi-scale representations due to the structure of their adopted ViTs is non-hierarchical. Meanwhile, they often only use the output features from the final layer for predicting results (i.e., ignoring the utilization of low-level features from the shallow layers) which may result in a certain degree of lacking multi-level representation learning ability. To address these issues, we propose a robust multi-stage tracker that effectively combines the advantages of both hierarchical and one-stream structured ViT as a tracking backbone to improve the multi-scale and multi-level representation learning abilities. Specifically, first of all, we design a hierarchical tracker with a three-stage backbone. In the first two stages of our tracker, we utilize a dual-branch structure to obtain multi-scale features of the template and search region separately. Especially, We design the local scale awareness modules based on simple MLP layers to capture multi-scale features. These modules remove complex operations such as convolutions or shifted window attentions, thus avoiding the performance degradation caused by traditional hierarchical ViTs. In the third stage (i.e. the main stage), we construct a global encoder based on the one-stream ViT to achieve efficient feature extraction and feature interaction for our tracker. Then, we design a multi-level feature integration module in the main stage to explicitly utilize the representation information learned from the shallow layers and fuse them with the features of the final layer to obtain multi-level representation information. Lastly, benefit from the these designs, our tracker can effectively capture more multi-scale and multi-level representations for robust tracking. Comprehensive experiments on GOT-10k, LaSOT, LaSOT$_{ext}$, TNL2K, UAV123, TrackingNet and VOT2020 benchmarks validate the effectiveness and robustness of our method.
- 11 June 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Emerging Topics in Computational Intelligence
- Vol. 10 (1) , 286-299
Towards Reliable Tracking: An Uncertainty-Aware Siamese Tracker
The recent years have witnessed the great advancement in accuracy made by Siamese trackers, while the performance is still limited by the insufficiency in robustness. Most of existing Siamese trackers regard their model output as deterministic results without explicitly estimating or optimizing the confidence level or reliability. As such, unreliable tracking results may be still utilized in model learning, and thus degrade the tracking model. In this paper, we introduce the notion of reliability into the tracking model, and propose a new tracking framework called Uncertainty-Aware Siamese Network to address this problem. Specifically, within the framework, we introduce the uncertainty awareness into the tracking model by designing a new network structure, which explicitly characterizes the confidence level about the tracker outputs in terms of regression accuracy and classification discriminability. In addition, with the uncertainty estimation module, we propose a new collaborative optimization model which aims to optimize tracking models under the constraint of reliability to make the tracking results as reliable as possible. Furthermore, based on the uncertainty estimation results, an online model updating scheme is developed to ensure both the adaptability and reliability of the model in dealing with appearance variations. Experiments performed on eight standard visual tracking benchmarks, including VOT2018, VOT2019, OTB100, NFS, UAV123, LaSOT, TrackingNet and Got-10 k, show that our uncertainty-aware Siamese tracker achieves state-of-the-art performance.
- 11 June 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Emerging Topics in Computational Intelligence
- Vol. 10 (1) , 286-299
Towards Reliable Tracking: An Uncertainty-Aware Siamese Tracker
The recent years have witnessed the great advancement in accuracy made by Siamese trackers, while the performance is still limited by the insufficiency in robustness. Most of existing Siamese trackers regard their model output as deterministic results without explicitly estimating or optimizing the confidence level or reliability. As such, unreliable tracking results may be still utilized in model learning, and thus degrade the tracking model. In this paper, we introduce the notion of reliability into the tracking model, and propose a new tracking framework called Uncertainty-Aware Siamese Network to address this problem. Specifically, within the framework, we introduce the uncertainty awareness into the tracking model by designing a new network structure, which explicitly characterizes the confidence level about the tracker outputs in terms of regression accuracy and classification discriminability. In addition, with the uncertainty estimation module, we propose a new collaborative optimization model which aims to optimize tracking models under the constraint of reliability to make the tracking results as reliable as possible. Furthermore, based on the uncertainty estimation results, an online model updating scheme is developed to ensure both the adaptability and reliability of the model in dealing with appearance variations. Experiments performed on eight standard visual tracking benchmarks, including VOT2018, VOT2019, OTB100, NFS, UAV123, LaSOT, TrackingNet and Got-10 k, show that our uncertainty-aware Siamese tracker achieves state-of-the-art performance.
- 8 April 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Artificial Intelligence
- Vol. 6 (10) , 2819-2829
Adaptive Perception for Unified Visual Multimodal Object Tracking
Recently, many multi-modal trackers prioritize RGB as the dominant modality, treating other modalities as auxiliary, and fine-tuning separately various multi-modal tasks. This imbalance in modality dependence limits the ability of methods to dynamically utilize complementary information from each modality in complex scenarios, making it challenging to fully perceive the advantages of multi-modal. As a result, a unified parameter model often underperforms in various multi-modal tracking tasks. To address this issue, we propose APTrack, a novel unified tracker designed for multi-modal adaptive perception. Unlike previous methods, APTrack explores a unified representation through an equal modeling strategy. This strategy allows the model to dynamically adapt to various modalities and tasks without requiring additional fine-tuning between different tasks. Moreover, our tracker integrates an adaptive modality interaction (AMI) module that efficiently bridges cross-modality interactions by generating learnable tokens. Experiments conducted on five diverse multi-modal datasets (RGBT234, LasHeR, VisEvent, DepthTrack, and VOT-RGBD2022) demonstrate that APTrack not only surpasses existing state-of-the-art unified multi-modal trackers but also outperforms trackers designed for specific multi-modal tasks.
- 8 April 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Artificial Intelligence
- Vol. 6 (10) , 2819-2829
Adaptive Perception for Unified Visual Multimodal Object Tracking
Recently, many multi-modal trackers prioritize RGB as the dominant modality, treating other modalities as auxiliary, and fine-tuning separately various multi-modal tasks. This imbalance in modality dependence limits the ability of methods to dynamically utilize complementary information from each modality in complex scenarios, making it challenging to fully perceive the advantages of multi-modal. As a result, a unified parameter model often underperforms in various multi-modal tracking tasks. To address this issue, we propose APTrack, a novel unified tracker designed for multi-modal adaptive perception. Unlike previous methods, APTrack explores a unified representation through an equal modeling strategy. This strategy allows the model to dynamically adapt to various modalities and tasks without requiring additional fine-tuning between different tasks. Moreover, our tracker integrates an adaptive modality interaction (AMI) module that efficiently bridges cross-modality interactions by generating learnable tokens. Experiments conducted on five diverse multi-modal datasets (RGBT234, LasHeR, VisEvent, DepthTrack, and VOT-RGBD2022) demonstrate that APTrack not only surpasses existing state-of-the-art unified multi-modal trackers but also outperforms trackers designed for specific multi-modal tasks.