Jie Zhang
Research Subjects
Research Interests
Jie Zhang has a total of 10 co-authors. Here are the top co-authors by number of shared publications:
- Aijuan Wang
College of Computer Science and Engineering, Chongqing University of Technology, Chongqing 400054, China
1 shared publication - Hai Nan
School of Computer Science and Engineering, Chongqing University of Technology, Chongqing, China
1 shared publication - Jiqiao Jiang
College of Computer Science and Engineering, Chongqing University of Technology, Chongqing 400054, China
1 shared publication - Meng Liu
School of Computer Science and Technology, Shandong Jianzhu University, Jinan, China
1 shared publication Qilang Ye
VCIP & TMCC & DISSec College of Computer Science & College of Cryptology and Cyber Science Nankai University
1 shared publication- Ran Liu
College of Computer Science, Chongqing University, Chongqing, China
1 shared publication
Latest Publications
- Open Access
When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion?
Can Multimodal Large Language Models (MLLMs) discern confused objects that are visually present but audio-absent? To study this, we introduce a new benchmark, AV-ConfuseBench, which simulates an ``Audio-Visual Confusion'' scene by modifying the corresponding sound of an object in the video, e.g., mute the sounding object and ask MLLMs Is there a/an muted-object sound''. Experimental results reveal that MLLMs, such as Qwen2.5-Omni and Gemini 2.5, struggle to discriminate non-existent audio due to visually dominated reasoning. Motivated by this observation, we introduce RL-CoMM, a Reinforcement Learning-based Collaborative Multi-MLLM that is built upon the Qwen2.5-Omni foundation. RL-CoMM includes two stages: 1) To alleviate visually dominated ambiguities, we introduce an external model, a Large Audio Language Model (LALM), as the reference model to generate audio-only reasoning. Then, we design a Step-wise Reasoning Reward function that enables MLLMs to self-improve audio-visual reasoning with the audio-only reference. 2) To ensure an accurate answer prediction, we introduce Answer-centered Confidence Optimization to reduce the uncertainty of potential heterogeneous reasoning differences. Extensive experiments on audio-visual question answering and audio-visual hallucination show that RL-CoMM improves the accuracy by 10~30\% over the baseline model with limited training data. Follow: https://github.com/rikeilong/AVConfusion.
Open AccessWhen Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion?
Can Multimodal Large Language Models (MLLMs) discern confused objects that are visually present but audio-absent? To study this, we introduce a new benchmark, AV-ConfuseBench, which simulates an ``Audio-Visual Confusion'' scene by modifying the corresponding sound of an object in the video, e.g., mute the sounding object and ask MLLMs Is there a/an muted-object sound''. Experimental results reveal that MLLMs, such as Qwen2.5-Omni and Gemini 2.5, struggle to discriminate non-existent audio due to visually dominated reasoning. Motivated by this observation, we introduce RL-CoMM, a Reinforcement Learning-based Collaborative Multi-MLLM that is built upon the Qwen2.5-Omni foundation. RL-CoMM includes two stages: 1) To alleviate visually dominated ambiguities, we introduce an external model, a Large Audio Language Model (LALM), as the reference model to generate audio-only reasoning. Then, we design a Step-wise Reasoning Reward function that enables MLLMs to self-improve audio-visual reasoning with the audio-only reference. 2) To ensure an accurate answer prediction, we introduce Answer-centered Confidence Optimization to reduce the uncertainty of potential heterogeneous reasoning differences. Extensive experiments on audio-visual question answering and audio-visual hallucination show that RL-CoMM improves the accuracy by 10~30\% over the baseline model with limited training data. Follow: https://github.com/rikeilong/AVConfusion.
- Open Access
- 2 September 2023
- Published by MDPI AG in Applied Sciences
- Vol. 13 (17) , 9945
Conversion between Number Systems in Membrane Computing
The number system is the representation method of numbers, and number system conversion is the most basic function of a computing system. The decimal system is the most commonly used number system; however, it may not necessarily be the most suitable number system in a computing system. This paper investigates the conversion methods between different number systems and designs corresponding cells, such as P systems, to implement them. The P systems we designed include Π10_2 and Π2_10 to implement conversion between decimal and binary, Π10_m and Πm_10 to implement conversion between decimal and m-base. The operation process of each P system was explained through examples, and their feasibility and effectiveness were verified through simulation software, UPSimulator.
- 2 September 2023
- Published by MDPI AG in Applied Sciences
- Vol. 13 (17) , 9945
Open AccessConversion between Number Systems in Membrane Computing
The number system is the representation method of numbers, and number system conversion is the most basic function of a computing system. The decimal system is the most commonly used number system; however, it may not necessarily be the most suitable number system in a computing system. This paper investigates the conversion methods between different number systems and designs corresponding cells, such as P systems, to implement them. The P systems we designed include Π10_2 and Π2_10 to implement conversion between decimal and binary, Π10_m and Πm_10 to implement conversion between decimal and m-base. The operation process of each P system was explained through examples, and their feasibility and effectiveness were verified through simulation software, UPSimulator.