Jiwei Zhang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Jiwei Zhang has a total of 7 co-authors. Here are the top co-authors by number of shared publications:
- Suhua Tang
The University of Electro-Communications, Tokyo, Japan
4 shared publications - Wei Li
School of Computer Science and Technology, Fudan University, Shanghai 200433, China
4 shared publications - Yi Yu
Hiroshima University, Japan
4 shared publications Haiyuan Wu
Sense Time Japan Co., Ltd. Kyoto Japan
2 shared publicationsHirotaka Hachiya
2 shared publications- Jianming Wu
Human-Centered AI Laboratories, KDDI Research, Inc, Fujimino, Japan
2 shared publications
Latest Publications
- Open Access
- 2 July 2025
- Published by Elsevier in Machine Learning with Applications
- Vol. 21
DCLMA: Deep correlation learning with multi-modal attention for visual-audio retrieval
- 2 July 2025
- Published by Elsevier in Machine Learning with Applications
- Vol. 21
Open AccessDCLMA: Deep correlation learning with multi-modal attention for visual-audio retrieval
- Open Access
- 5 May 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 82302-82311
Dialogue-to-Video Retrieval via Multi-Grained Attention Network
Dialogue-to-video retrieval is an interesting while challenging task, which exploits an AI agent to retrieve the video that matches and aligns with the conversational context between users. In particular, given a history of dialogue exchanges, the agent is expected to identify the most fitting video content that complements the ongoing conversation. The computational complexity posed by the processing of videos within deep neural networks encourages us to adapt CLIP, a cutting-edge large multi-modal models (LLMs), into the dialogue-video domain. On this basis, we propose a multi-grained attention network (MGAT), integrating query-scoring, dual-softmax and query-bank normalization techniques. We design a multi-grained attention module to optimize the inadequate modeling of conversation semantics in existing pre-trained models, dynamically assign weights during conversation feature extraction, and introduce conversation context features in multimodal alignment. Most importantly, fine-grained similarity by per-round query-frame scoring and coarse-grained similarity by high-level semantics of all rounds of dialogues for each video are respectively calculated, which are further learned via multi-grained attention mechanism. This approach effectively transfers CLIP’s text-image multimodal knowledge into the dialogue-video retrieval system, alleviating the need for resource-intensive and costly fine-tuning dialogue-video procedures. Extensive experiments on multiple datasets demonstrate that our method outperforms state-of-the-art approaches.
- 5 May 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 82302-82311
Open AccessDialogue-to-Video Retrieval via Multi-Grained Attention Network
Dialogue-to-video retrieval is an interesting while challenging task, which exploits an AI agent to retrieve the video that matches and aligns with the conversational context between users. In particular, given a history of dialogue exchanges, the agent is expected to identify the most fitting video content that complements the ongoing conversation. The computational complexity posed by the processing of videos within deep neural networks encourages us to adapt CLIP, a cutting-edge large multi-modal models (LLMs), into the dialogue-video domain. On this basis, we propose a multi-grained attention network (MGAT), integrating query-scoring, dual-softmax and query-bank normalization techniques. We design a multi-grained attention module to optimize the inadequate modeling of conversation semantics in existing pre-trained models, dynamically assign weights during conversation feature extraction, and introduce conversation context features in multimodal alignment. Most importantly, fine-grained similarity by per-round query-frame scoring and coarse-grained similarity by high-level semantics of all rounds of dialogues for each video are respectively calculated, which are further learned via multi-grained attention mechanism. This approach effectively transfers CLIP’s text-image multimodal knowledge into the dialogue-video retrieval system, alleviating the need for resource-intensive and costly fine-tuning dialogue-video procedures. Extensive experiments on multiple datasets demonstrate that our method outperforms state-of-the-art approaches.
- Open Access
- 28 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 39753-39764
Adversarial Contrastive Autoencoder With Shared Attention for Audio-Visual Correlation Learning
Cross-modal audio-visual correlation learning has been an active research topic, which aims to embed audio and visual feature sequences into a common subspace where their correlation is maximized. The challenge of audio-visual correlation learning lies in two major aspects: i) audio and visual feature sequences respectively contain different patterns belonging to different feature spaces, and ii) semantic mismatch between audio and visual sequences inevitably happens during cross-modal matching. Most existing methods only take the first aspect into account, therefore facing the difficulty in distinguishing matched and mismatched semantic correlations between audio and visual sequences. In this work, an adversarial contrastive autoencoder with a shared attention network (ACASA) is proposed for correlation learning in audio-visual retrieval. In particular, the proposed shared attention mechanism is parameterized, in which local salient information is enhanced to contribute to the final feature representation. Simultaneously, adversarial contrastive learning is exploited to maximize semantic feature consistency and improve the ability to distinguish matched and mismatched samples. Both inter-modal and intra-modal semantic information are utilized to supervise the model to learn more discriminative feature representation. Extensive experiments on the VEGAS and AVE datasets demonstrate that the proposed ACASA method outperforms state-of-the-art approaches in cross-modal audio-visual retrieval.
- 28 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 39753-39764
Open AccessAdversarial Contrastive Autoencoder With Shared Attention for Audio-Visual Correlation Learning
Cross-modal audio-visual correlation learning has been an active research topic, which aims to embed audio and visual feature sequences into a common subspace where their correlation is maximized. The challenge of audio-visual correlation learning lies in two major aspects: i) audio and visual feature sequences respectively contain different patterns belonging to different feature spaces, and ii) semantic mismatch between audio and visual sequences inevitably happens during cross-modal matching. Most existing methods only take the first aspect into account, therefore facing the difficulty in distinguishing matched and mismatched semantic correlations between audio and visual sequences. In this work, an adversarial contrastive autoencoder with a shared attention network (ACASA) is proposed for correlation learning in audio-visual retrieval. In particular, the proposed shared attention mechanism is parameterized, in which local salient information is enhanced to contribute to the final feature representation. Simultaneously, adversarial contrastive learning is exploited to maximize semantic feature consistency and improve the ability to distinguish matched and mismatched samples. Both inter-modal and intra-modal semantic information are utilized to supervise the model to learn more discriminative feature representation. Extensive experiments on the VEGAS and AVE datasets demonstrate that the proposed ACASA method outperforms state-of-the-art approaches in cross-modal audio-visual retrieval.
- 19 July 2023
- Published by Springer Nature in Neural Computing and Applications
- Vol. 35 (27) , 20173-20187
Multi-scale network with shared cross-attention for audio–visual correlation learning
- 19 July 2023
- Published by Springer Nature in Neural Computing and Applications
- Vol. 35 (27) , 20173-20187
Multi-scale network with shared cross-attention for audio–visual correlation learning
- Open Access
- 24 February 2023
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 19 (3s) , 1-21
Variational Autoencoder with CCA for Audio–Visual Cross-modal Retrieval
Cross-modal retrieval is to utilize one modality as a query to retrieve data from another modality, which has become a popular topic in information retrieval, machine learning, and databases. Finding a method to effectively measure the similarity between different modality data is the major challenge of cross-modal retrieval. Although several research works have calculated the correlation between different modality data via learning a common subspace representation, the encoder’s ability to extract features from multi-modal information is not satisfactory. In this article, we present a novel variational autoencoder architecture for audio–visual cross-modal retrieval by learning paired audio–visual correlation embedding and category correlation embedding as constraints to reinforce the mutuality of audio–visual information. On the one hand, audio encoder and visual encoder separately encode audio data and visual data into two different latent spaces. Further, two mutual latent spaces are respectively constructed by canonical correlation analysis. On the other hand, probabilistic modeling methods are used to deal with possible noise and missing information in the data. Additionally, in this way, the cross-modal discrepancies from intra-modal and inter-modal information are simultaneously eliminated in the joint embedding subspace. We conduct extensive experiments over two benchmark datasets. The experimental results confirm that the proposed architecture is effective in learning audio–visual correlation and is appreciably better than the existing cross-modal retrieval methods.
- 24 February 2023
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 19 (3s) , 1-21
Open AccessVariational Autoencoder with CCA for Audio–Visual Cross-modal Retrieval
Cross-modal retrieval is to utilize one modality as a query to retrieve data from another modality, which has become a popular topic in information retrieval, machine learning, and databases. Finding a method to effectively measure the similarity between different modality data is the major challenge of cross-modal retrieval. Although several research works have calculated the correlation between different modality data via learning a common subspace representation, the encoder’s ability to extract features from multi-modal information is not satisfactory. In this article, we present a novel variational autoencoder architecture for audio–visual cross-modal retrieval by learning paired audio–visual correlation embedding and category correlation embedding as constraints to reinforce the mutuality of audio–visual information. On the one hand, audio encoder and visual encoder separately encode audio data and visual data into two different latent spaces. Further, two mutual latent spaces are respectively constructed by canonical correlation analysis. On the other hand, probabilistic modeling methods are used to deal with possible noise and missing information in the data. Additionally, in this way, the cross-modal discrepancies from intra-modal and inter-modal information are simultaneously eliminated in the joint embedding subspace. We conduct extensive experiments over two benchmark datasets. The experimental results confirm that the proposed architecture is effective in learning audio–visual correlation and is appreciably better than the existing cross-modal retrieval methods.
- 19 February 2023
- Published by Wiley in Computer Animation and Virtual Worlds
- Vol. 34 (6)
Multi‐feature subspace representation network for person re‐identification via bird's‐eye view image
Person re‐identification (Re‐ID) is one of the most remarkable research topics that widely applied in our daily lives. For person Re‐ID in bird's eye scenes, traditional computer vision‐based methods used multiple features, for example, texture and color, of a pedestrian's head and shoulders. Those methods are difficult to cope with environments of variety and the change the appearance of different people due to the instability of feature detection. On the other hand, although recent advanced deep learning‐based methods are powerful to extract discriminative features, the requirement of a large amount of annotated training data restricts the appliable tasks. To overcome this problem, in this article, we propose a novel method fusing multiple heterogeneous features through a multi‐feature subspace representation network (MFSRN) to maximize the classification performance while keeping the disparity among features as small as possible, that is, common‐subspace constraints. We conducted comparative experiments with state‐of‐the‐art models on the bird's‐eye view person dataset, and extensive experimental results demonstrated that our proposed MFSRN could achieve better recognition performance. Furthermore, the validity and stability of the method are confirmed.
- 19 February 2023
- Published by Wiley in Computer Animation and Virtual Worlds
- Vol. 34 (6)
Multi‐feature subspace representation network for person re‐identification via bird's‐eye view image
Person re‐identification (Re‐ID) is one of the most remarkable research topics that widely applied in our daily lives. For person Re‐ID in bird's eye scenes, traditional computer vision‐based methods used multiple features, for example, texture and color, of a pedestrian's head and shoulders. Those methods are difficult to cope with environments of variety and the change the appearance of different people due to the instability of feature detection. On the other hand, although recent advanced deep learning‐based methods are powerful to extract discriminative features, the requirement of a large amount of annotated training data restricts the appliable tasks. To overcome this problem, in this article, we propose a novel method fusing multiple heterogeneous features through a multi‐feature subspace representation network (MFSRN) to maximize the classification performance while keeping the disparity among features as small as possible, that is, common‐subspace constraints. We conducted comparative experiments with state‐of‐the‐art models on the bird's‐eye view person dataset, and extensive experimental results demonstrated that our proposed MFSRN could achieve better recognition performance. Furthermore, the validity and stability of the method are confirmed.
- 8 August 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 259-263
A Super Baseline for Pedestrian Re-Identification
In this paper, we propose a new baseline that combines and improves two types of deep learning algorithms to achieve more accurate pedestrian re-identification from images taken by surveillance cameras. We introduce some tricks into a famous strong baseline [1]. The main contributions are: 1) We improve IBN-Net [2] to replace the original ResNet50 (last stride = 1). 2) We perform model learning by using three common databases with different characteristics at the same time, except for the operation part of random erasing augmentation. 3) We add two new classifiers. 4) We optimize the parameters to prevent overfitting. We conducted ablation experiments on each trick using unlearned data and confirmed the effectiveness and stability of the proposed method from the results.
- 8 August 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 259-263
A Super Baseline for Pedestrian Re-Identification
In this paper, we propose a new baseline that combines and improves two types of deep learning algorithms to achieve more accurate pedestrian re-identification from images taken by surveillance cameras. We introduce some tricks into a famous strong baseline [1]. The main contributions are: 1) We improve IBN-Net [2] to replace the original ResNet50 (last stride = 1). 2) We perform model learning by using three common databases with different characteristics at the same time, except for the operation part of random erasing augmentation. 3) We add two new classifiers. 4) We optimize the parameters to prevent overfitting. We conducted ablation experiments on each trick using unlearned data and confirmed the effectiveness and stability of the proposed method from the results.