JZJiwei Zhang

Jiwei Zhang

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Jiwei Zhang has a total of 7 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Dialogue-to-Video Retrieval via Multi-Grained Attention Network

    by Yi Yu, Jiwei Zhang, Suhua Tang, Wei Li

    Dialogue-to-video retrieval is an interesting while challenging task, which exploits an AI agent to retrieve the video that matches and aligns with the conversational context between users. In particular, given a history of dialogue exchanges, the agent is expected to identify the most fitting video content that complements the ongoing conversation. The computational complexity posed by the processing of videos within deep neural networks encourages us to adapt CLIP, a cutting-edge large multi-modal models (LLMs), into the dialogue-video domain. On this basis, we propose a multi-grained attention network (MGAT), integrating query-scoring, dual-softmax and query-bank normalization techniques. We design a multi-grained attention module to optimize the inadequate modeling of conversation semantics in existing pre-trained models, dynamically assign weights during conversation feature extraction, and introduce conversation context features in multimodal alignment. Most importantly, fine-grained similarity by per-round query-frame scoring and coarse-grained similarity by high-level semantics of all rounds of dialogues for each video are respectively calculated, which are further learned via multi-grained attention mechanism. This approach effectively transfers CLIP’s text-image multimodal knowledge into the dialogue-video retrieval system, alleviating the need for resource-intensive and costly fine-tuning dialogue-video procedures. Extensive experiments on multiple datasets demonstrate that our method outperforms state-of-the-art approaches.

  • Open Access

    Adversarial Contrastive Autoencoder With Shared Attention for Audio-Visual Correlation Learning

    by Jiwei Zhang, Yi Yu, Suhua Tang, Wei Li

    Cross-modal audio-visual correlation learning has been an active research topic, which aims to embed audio and visual feature sequences into a common subspace where their correlation is maximized. The challenge of audio-visual correlation learning lies in two major aspects: i) audio and visual feature sequences respectively contain different patterns belonging to different feature spaces, and ii) semantic mismatch between audio and visual sequences inevitably happens during cross-modal matching. Most existing methods only take the first aspect into account, therefore facing the difficulty in distinguishing matched and mismatched semantic correlations between audio and visual sequences. In this work, an adversarial contrastive autoencoder with a shared attention network (ACASA) is proposed for correlation learning in audio-visual retrieval. In particular, the proposed shared attention mechanism is parameterized, in which local salient information is enhanced to contribute to the final feature representation. Simultaneously, adversarial contrastive learning is exploited to maximize semantic feature consistency and improve the ability to distinguish matched and mismatched samples. Both inter-modal and intra-modal semantic information are utilized to supervise the model to learn more discriminative feature representation. Extensive experiments on the VEGAS and AVE datasets demonstrate that the proposed ACASA method outperforms state-of-the-art approaches in cross-modal audio-visual retrieval.


  • Open Access

    Variational Autoencoder with CCA for Audio–Visual Cross-modal Retrieval

    by Jiwei Zhang, Yi Yu, Suhua Tang, Jianming Wu, Wei Li

    Cross-modal retrieval is to utilize one modality as a query to retrieve data from another modality, which has become a popular topic in information retrieval, machine learning, and databases. Finding a method to effectively measure the similarity between different modality data is the major challenge of cross-modal retrieval. Although several research works have calculated the correlation between different modality data via learning a common subspace representation, the encoder’s ability to extract features from multi-modal information is not satisfactory. In this article, we present a novel variational autoencoder architecture for audio–visual cross-modal retrieval by learning paired audio–visual correlation embedding and category correlation embedding as constraints to reinforce the mutuality of audio–visual information. On the one hand, audio encoder and visual encoder separately encode audio data and visual data into two different latent spaces. Further, two mutual latent spaces are respectively constructed by canonical correlation analysis. On the other hand, probabilistic modeling methods are used to deal with possible noise and missing information in the data. Additionally, in this way, the cross-modal discrepancies from intra-modal and inter-modal information are simultaneously eliminated in the joint embedding subspace. We conduct extensive experiments over two benchmark datasets. The experimental results confirm that the proposed architecture is effective in learning audio–visual correlation and is appreciably better than the existing cross-modal retrieval methods.


  • Multi‐feature subspace representation network for person re‐identification via bird's‐eye view image

    by Jiwei Zhang, Haiyuan Wu, Qian Chen, Hirotaka Hachiya

    Person re‐identification (Re‐ID) is one of the most remarkable research topics that widely applied in our daily lives. For person Re‐ID in bird's eye scenes, traditional computer vision‐based methods used multiple features, for example, texture and color, of a pedestrian's head and shoulders. Those methods are difficult to cope with environments of variety and the change the appearance of different people due to the instability of feature detection. On the other hand, although recent advanced deep learning‐based methods are powerful to extract discriminative features, the requirement of a large amount of annotated training data restricts the appliable tasks. To overcome this problem, in this article, we propose a novel method fusing multiple heterogeneous features through a multi‐feature subspace representation network (MFSRN) to maximize the classification performance while keeping the disparity among features as small as possible, that is, common‐subspace constraints. We conducted comparative experiments with state‐of‐the‐art models on the bird's‐eye view person dataset, and extensive experimental results demonstrated that our proposed MFSRN could achieve better recognition performance. Furthermore, the validity and stability of the method are confirmed.

  • A Super Baseline for Pedestrian Re-Identification

    by Jiwei Zhang, Haiyuan Wu

    In this paper, we propose a new baseline that combines and improves two types of deep learning algorithms to achieve more accurate pedestrian re-identification from images taken by surveillance cameras. We introduce some tricks into a famous strong baseline [1]. The main contributions are: 1) We improve IBN-Net [2] to replace the original ResNet50 (last stride = 1). 2) We perform model learning by using three common databases with different characteristics at the same time, except for the operation part of random erasing augmentation. 3) We add two new classifiers. 4) We optimize the parameters to prevent overfitting. We conducted ablation experiments on each trick using unlearned data and confirmed the effectiveness and stability of the proposed method from the results.