Cross-Modal Uncertainty Modeling With Diffusion-Based Refinement for Text-Based Person Retrieval
- 29 October 2024
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 35 (3) , 2881-2893
Abstract
Text-based person retrieval (TBPR) is a challenging task that aims at retrieving candidate pedestrian images from a gallery, using textual descriptions as queries. Existing methods generally assume that the textual query and the unique candidate image have a certain cross-modal relationship under one-to-one constraint, and optimize their conditional probability via a discriminative paradigm. However, in real scenarios of TBPR, a textual query may associate with multiple candidate images at one time, indicating that the uncertainty resides in the one-to-many cross-modal relationship. Moreover, the learnt conditional probabilities from the discriminative paradigm of existing methods may be less effective in reflecting the joint probabilities of the textual query and candidate images. To tackle these problems, we propose a novel method termed Cross-modal Uncertainty Modeling with Diffusion-based Refinement (CUMDR) for the TBPR task. First, we implicitly model the cross-modal uncertainty to capture richer semantics and complex correlations, thus generating diverse yet plausible retrievals. Additionally, to reasonably mitigate the impact of noisy data with high uncertainty, we quantify the uncertainty to allocate the importance of raw and complement annotations, which are generated from the multi-modal large language model based on the retrieval-augmented template. Finally, we propose a novel diffusion-based denoiser to progressively refine cross-modal alignments by learning joint probabilities. Extensive experiments on three TBPR datasets demonstrate the superior performance and generalizability of our CUMDR approach compared to the latest methods. Our anonymous implementation repository is available at https://github.com/Shenshen7/CUMDR.Keywords
Funding Information
- National Natural Science Foundation of China (62222203)
- National Natural Science Foundation of China (62072080)
- New Cornerstone Science Foundation through the XPLORER PRIZE
- Sichuan Science and Technology Program (2023-XT00-00001-GX)
This publication has 41 references indexed in Scilit:
- Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval2023
- A Simple and Robust Correlation Filtering Method for Text-Based Person Search2022
- Vector Quantized Diffusion Model for Text-to-Image Synthesis2022
- Probabilistic Embeddings for Cross-Modal Retrieval2021
- View-Invariant Probabilistic Embedding for Human Pose2020
- ViTAA: Visual-Textual Attributes Alignment in Person Search by Natural Language2020
- Data Uncertainty Learning in Face Recognition2020
- Unsupervised Person Re-Identification via Multi-Label Classification2020
- Deep Cross-Modal Projection Learning for Image-Text Matching2018
- Person Search with Natural Language Description2017