Yangliu Kuai

Sequence learning based tracking frameworks are popular in the tracking community. In practice, its auto-regressive sequence generation manner leads to inferior performance and high latency compared with latest advanced trackers. In this paper, to mitigate this issue, we propose an efficient and effective sequence-to-sequence tracking framework named FastSeqTrack. FastSeqTrack differs from previous sequence learning based trackers in terms of token initialization and sequence generation manner. Four tracking tokens are appended to patch embeddings and generated in the encoder as initial guesses for the bounding box sequence, which improves the tracking accuracy compared with randomly initialized tokens. Tracking tokens are then parallelly fed into the decoder in a one-pass manner and greatly boost the forward inference speed compared with the auto-regressive manner. Inspired by the early-exit mechanism, we inject internal classifiers after each decoder layer to early terminate forward inference when the softmax confidence is sufficiently reliable. In easy tracking frames, early exits avoid network overthinking and unnecessary computation. Extensive experiments on multiple benchmarks demonstrate that FastSeqTrack runs over 100 fps and showcases superior performance against state-of-the-art trackers. Codes and models are available at https://github.com/vision4drones/FastSeqTrack.

With the extensive use of multi-sensors in Unmanned Aerial Vehicles (UAVs), multi-modality information processing has become the research focus. In academic research pertaining to object detection and tracking tasks in unmanned aerial vehicles, researchers often align visible-infrared image pairs as a preprocessing step. However, in actual tasks, the dual-modality image pair acquired by unmanned aerial vehicles is unaligned, which significantly limits the application of downstream tasks. At present, there are no publicly available multi-modality image alignment datasets for unmanned aerial vehicles. In this paper, we present a large-scale benchmark for the dual-modality image alignment task in unmanned aerial vehicles, including 81,000 training image pairs and 15,000 testing image pairs. Meanwhile, we propose a transformer-based dual-modality image alignment network as the baseline for this benchmark. Firstly, the algorithm extracts multi-scale features for image representation to address unaligned image pairs with varying resolutions. Secondly, a transformer-based alignment network is proposed to improve the fusion of features from heterogeneous modalities. Finally, deformable attention is adopted to alleviate the problem of memory explosion. Numerous experiments on this dual-modality image alignment benchmark are conducted to demonstrate the effectiveness of our algorithm. Source codes are available at https://github.com/gaozhinanjiu/UAVmatch.

The use of unmanned aerial vehicles (UAVs) for visible–thermal object detection has emerged as a powerful technique to improve accuracy and resilience in challenging contexts, including dim lighting and severe weather conditions. However, most existing research relies on Convolutional Neural Network (CNN) frameworks, limiting the application of the Transformer’s attention mechanism to mere fusion modules and neglecting its potential for comprehensive global feature modeling. In response to this limitation, this study introduces an innovative dual-modal object detection framework called Visual Prompt multi-modal Detection (VIP-Det) that harnesses the Transformer architecture as the primary feature extractor and integrates vision prompts for refined feature fusion. Our approach begins with the training of a single-modal baseline model to solidify robust model representations, which is then refined through fine-tuning that incorporates additional modal data and prompts. Tests on the DroneVehicle dataset show that our algorithm achieves remarkable accuracy, outperforming comparable Transformer-based methods. These findings indicate that our proposed methodology marks a significant advancement in the realm of UAV-based object detection, holding significant promise for enhancing autonomous surveillance and monitoring capabilities in varied and challenging environments.

This paper investigates how to implement accurate RGB-T tracking by achieving effective feature enhancement of the target and adaptive fusion of the complementary information in RGB and thermal infrared modalities. Inspired by the excellent long-range dependency modeling ability of transformer, we propose a novel RGBT tracking method based on transformer via progressive feature enhancement and fusion. The overall flowchart of our proposed tracker consists of a two-branch Siamese network, respectively an exemplar branch and a search branch. Firstly, deep features of the RGB and thermal infrared images are extracted by a backbone. And then the features in each branch are enhanced progressively in the channel and spatial dimensions. Specifically, in the channel dimension, channel attention feature module (CAFM) is designed to adaptively enhance the RGB and thermal infrared features. And in the spatial dimension, transformer self-attention mechanism with AiA module is integrated to enhance the dual-modality features. Next, the enhanced features from the exemplar and search branches are fused based on the transformer cross-attention mechanism, which can achieve global and deep interaction between the exemplar and search images. Finally, the fused features are fed into a corner predictor head to estimate the target state. Experiments on two widely used public benchmarks (RGBT234 and LasHeR) demonstrate the effectiveness and efficiency of our proposed method when compared to many other state-of-the-art (SOTA) trackers released recently.

Assessing the detection capability is crucial for the design and implementation of distributed MIMO radar systems. However, existing MIMO radar models have mainly been established under ideal conditions, assuming spatially uncorrelated target echoes, strictly orthogonal transmitted waveforms, and negligible system errors. Such idealized assumptions may not hold in practical scenarios, making it challenging to comprehensively and realistically evaluate radar detection performance. Motivated by this, our paper investigates the theoretical radar detection capability by removing these idealized assumptions. We develop a more general system model for distributed MIMO radar that considers the probable spatial correlations of target echoes, waveform non-orthogonality, and various system errors. These errors include synchronization errors in time, frequency, and phase, uncertainties in site position and beam pointing, as well as mismatch errors between the test-cell center and the target. Using this enhanced model, we derive an accurate and efficient equation to evaluate detection performance by employing a moment-matching approximation strategy. Additionally, we analyze the impact of non-ideal conditions through simulations. Our findings reveal several important insights. Firstly, channel-correlated target returns can be advantageous for target detection, particularly in low signal-to-noise ratio (SNR) scenarios. Secondly, non-orthogonal waveforms increase the sensitivity of radar detection performance to non-ideal factors. Lastly, the threshold at which system errors significantly degrade radar detection performance can be adjusted by radar parameters such as waveform characteristics and beamwidths. These results contribute to the design of radar systems in various aspects, including site placement, waveform optimization, beamforming, and test-cell creation.

Evaluating target measurement performance is significant for the design and application of distributed multiple-input multiple-output (MIMO) radar. Existing research into the target measurement performance was mostly conducted under ideal conditions. In practice, however, distributed MIMO radar may inevitably suffer from the system errors in time sync, frequency sync, and beam pointing as well as the uncertainties in sensor positions. These non-ideal factors may lead to a degradation of the target estimation performance to some extent, and this joint impact remains unclear. Motivated by this, in this paper, we systematically explore the target measurement performance of distributed MIMO radar under non-ideal conditions. First, we formulate a general system model for distributed MIMO radar incorporating the errors in time sync, frequency sync, sensor position, and beam pointing. By treating these system errors as Gaussian random variables, we form the hybrid Cramér-Rao Bound (HCRB) for the joint estimation of the target parameters and these errors. The derived HCRB is shown to be effective in lower bounding the mean-square error (MSE) of the hybrid maximum likelihood/maximum a posteriori (ML/MAP) estimate. Therefore, this bound can be used as an asymptotic benchmark or metric of the radar performance in target measurement. Based on the HCRB, we also examine the effects of these non-ideal factors on the target measurement performance by simulations. The results offer some perspective on the relationships between the system tolerances for these system errors and the radar parameters; these findings will be helpful in the system design of distributed MIMO radar.

In the field of drone-based object tracking, utilization of the infrared modality can improve the robustness of the tracker in scenes with severe illumination change and occlusions and expand the applicable scene of the drone object tracking task. Inspired by the great achievements of Transformer structure in the field of RGB object tracking, we design a dual-modality object tracking network based on Transformer. To better address the problem of visible-infrared information fusion, we propose a Dual-Feature Aggregation Network that utilizes attention mechanisms in both spatial and channel dimensions to aggregate heterogeneous modality feature information. The proposed algorithm has achieved better performance by comparing with the mainstream algorithms in the drone-based dual-modality object tracking dataset VTUAV. Additionally, the algorithm is lightweight and can be easily deployed and executed on a drone edge computing platform. In summary, the proposed algorithm is mainly applicable to the field of drone dual-modality object tracking and the algorithm is optimized so that it can be deployed on the drone edge computing platform. The effectiveness of the algorithm is proved by experiments and the scope of drone object tracking is extended effectively.