Deeply Coupled Convolution–Transformer With Spatial–Temporal Complementary Learning for Video-Based Person Re-Identification
- 26 May 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 35 (10) , 13753-13763
Abstract
Advanced deep convolutional neural networks (CNNs) have shown great success in video-based person re-identification (Re-ID). However, they usually focus on the most obvious regions of persons with a limited global representation ability. Recently, it witnesses that Transformers explore the interpatch relationships with global observations for performance improvements. In this work, we take both the sides and propose a novel spatial–temporal complementary learning framework named deeply coupled convolution–transformer (DCCT) for high-performance video-based person Re-ID. First, we couple CNNs and Transformers to extract two kinds of visual features and experimentally verify their complementarity. Furthermore, in spatial, we propose a complementary content attention (CCA) to take advantages of the coupled structure and guide independent features for spatial complementary learning. In temporal, a hierarchical temporal aggregation (HTA) is proposed to progressively capture the interframe dependencies and encode temporal information. Besides, a gated attention (GA) is used to deliver aggregated temporal information into the CNN and Transformer branches for temporal complementary learning. Finally, we introduce a self-distillation training strategy to transfer the superior spatial–temporal knowledge to backbone networks for higher accuracy and more efficiency. In this way, two kinds of typical features from same videos are integrated mechanically for more informative representations. Extensive experiments on four public Re-ID benchmarks demonstrate that our framework could attain better performances than most state-of-the-art methods.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62101092, 62293542, U1903215)
- Chinese Association for Artificial Intelligence-Huawei MindSpore Open Fund (CAAIXSJLJJ-2021-067A)
- National Key Research and Development Program of China (2018AAA0102001)
- Fundamental Research Funds for the Central Universities (DUT22ZD210)
This publication has 66 references indexed in Scilit:
- Jointly Attentive Spatial-Temporal Pooling Networks for Video-Based Person Re-identification2017
- Unlabeled Samples Generated by GAN Improve the Person Re-identification Baseline in Vitro2017
- Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization2017
- See the Forest for the Trees: Joint Spatial and Temporal Recurrent Neural Networks for Video-Based Person Re-identification2017
- Joint Detection and Identification Feature Learning for Person Search2017
- MARS: A Video Benchmark for Large-Scale Person Re-Identification2016
- Deep Residual Learning for Image Recognition2016
- Person Re-identification by Multi-Channel Parts-Based CNN with Improved Triplet Loss Function2016
- Deep feature learning with relative distance comparison for person re-identificationPattern Recognition, 2015
- Large-Scale Machine Learning with Stochastic Gradient Descent2010