Asymmetric Cross-Scale Alignment for Text-Based Person Search
- 5 December 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 25 (15209210) , 7699-7709
Abstract
Text-based person search (TBPS) is of significant importance in intelligent surveillance, which aims to retrieve pedestrian images with high semantic relevance to a given text description. This retrieval task is characterized with both modal heterogeneity and fine-grained matching. To implement this task, one needs to extract multi-scale features from both image and text domains, and then perform the cross-modal alignment. However, most existing approaches only consider the alignment confined at their individual scales, e.g., an image-sentence or a region-phrase scale. Such a strategy adopts the presumable alignment in feature extraction, while overlooking the cross-scale alignment, e.g., image-phrase. In this paper, we present a transformer-based model to extract multi-scale representations, and perform Asymmetric Cross-Scale Alignment (ACSA) to precisely align the two modalities. Specifically, ACSA consists of a global-level alignment module and an asymmetric cross-attention module, where the former aligns an image and texts on a global scale, and the latter applies the cross-attention mechanism to dynamically align the cross-modal entities in region/image-phrase scales. Extensive experiments on two benchmark datasets CUHK-PEDES and RSTPReid demonstrate the effectiveness of our approach. Codes are available at https://github.com/mul-hjh/ACSA .Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62176178, U19A2073, 62002096)
- Natural Science Foundation of Tianjin Municipality (19JCYBJC16000)
This publication has 42 references indexed in Scilit:
- Language Person Search with Mutually Connected Classification Loss2019
- Deep Cross-Modal Projection Learning for Image-Text Matching2018
- Improving Deep Visual Representation for Person Re-identification by Global and Local Image-language Association2018
- Stacked Cross Attention for Image-Text Matching2018
- What-and-where to match: Deep spatially multiplicative integration networks for person re-identificationPattern Recognition, 2018
- Improving Text-Based Person Search by Spatial Matching and Adaptive Threshold2018
- Person Search with Natural Language Description2017
- Deep Residual Learning for Image Recognition2016
- Speech recognition with deep recurrent neural networks2013
- NLTK2004