Jing Zhang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Jing Zhang has a total of 7 co-authors. Here are the top co-authors by number of shared publications:
- Dan Guo
Hefei University of Technology, Hefei, China
4 shared publications - Meng Wang
Hefei University of Technology, Hefei, China
3 shared publications - Peipei Song
University of Science and Technology of China, Hefei, China
2 shared publications - Jiahui Sun
Hefei University of Technology, School of Computer Science and Information Engineering, China
1 shared publication - Liang Zheng
Australian National University
1 shared publication - Xun Yang
University of Science and Technology of China, Hefei, China
1 shared publication
Latest Publications
- Open Access
- 6 December 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-7
EmoSEM: Segment and Explain Emotion Stimuli in Visual Art
- 6 December 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-7
Open AccessEmoSEM: Segment and Explain Emotion Stimuli in Visual Art
- 9 November 2024
- Published by Springer Nature
- p. 397-413
Training A Small Emotional Vision Language Model for Visual Art Comprehension
- 9 November 2024
- Published by Springer Nature
- p. 397-413
Training A Small Emotional Vision Language Model for Visual Art Comprehension
- Open Access
- 26 April 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-7
Syntax-Controllable Video Captioning with Tree-Structural Syntax Augmentation
Traditional video captioning tasks often suffer from uncontrollable syntax, limiting their adaptability to diverse practical requirements. This paper explores a challenging task known as syntax-controllable video captioning (SCVC). Specifically, this task aims to generate captions for videos that convey the visual content accurately and adhere to the syntactic structure provided in an example sentence. To tackle the SCVC task, we propose a novel Tree-Structural Syntax Augmentation Network (TSAN). It consists of a tree-structural syntax encoder to capture syntactic parent-child relationships of example sentences and a syntax-customized caption generator to integrate syntax and semantic hints effectively. Moreover, we introduce a syntax-semantic joint optimization strategy that considers both syntax and semantic constraints for joint training. Considering the fact that there are no available syntax-controllable video caption datasets, we use the traditional video captioning dataset to train the model. In the inference process, the model generates syntax-controllable captions in a zero-shot manner, applying exemplar sentences randomly selected from an open corpus and unseen in the training data. The experiments on MSRVTT and VATEX datasets validate the superior performance of TSAN in both semantic and syntactic evaluations. Extensive ablation studies and visualizations further show the effectiveness of the proposed method.
- 26 April 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-7
Open AccessSyntax-Controllable Video Captioning with Tree-Structural Syntax Augmentation
Traditional video captioning tasks often suffer from uncontrollable syntax, limiting their adaptability to diverse practical requirements. This paper explores a challenging task known as syntax-controllable video captioning (SCVC). Specifically, this task aims to generate captions for videos that convey the visual content accurately and adhere to the syntactic structure provided in an example sentence. To tackle the SCVC task, we propose a novel Tree-Structural Syntax Augmentation Network (TSAN). It consists of a tree-structural syntax encoder to capture syntactic parent-child relationships of example sentences and a syntax-customized caption generator to integrate syntax and semantic hints effectively. Moreover, we introduce a syntax-semantic joint optimization strategy that considers both syntax and semantic constraints for joint training. Considering the fact that there are no available syntax-controllable video caption datasets, we use the traditional video captioning dataset to train the model. In the inference process, the model generates syntax-controllable captions in a zero-shot manner, applying exemplar sentences randomly selected from an open corpus and unseen in the training data. The experiments on MSRVTT and VATEX datasets validate the superior performance of TSAN in both semantic and syntactic evaluations. Extensive ablation studies and visualizations further show the effectiveness of the proposed method.
- Open Access
- 11 January 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 20 (4) , 1-23
Visual-linguistic-stylistic Triple Reward for Cross-lingual Image Captioning
Generating image captions in different languages is worth exploring and essential for non-native speakers. Nevertheless, collecting paired annotation for every language is time-consuming and impractical, particularly for minor languages. To this end, the cross-lingual image captioning task is proposed, which leverages existing image-source caption annotation data and wild unrelated target corpus to generate satisfactory caption in the target language. Current methods perform a two-step translation process of image-to-pivot (source) and pivot-to-target. The distinct two-step process comes with certain caption issues, such as the weak semantic alignment between the image and the generated caption and the generated caption’s non-target language style. To address these issues, we propose an end-to-end reinforce learning framework with Visual-Linguistic-Stylistic Triple Reward named TriR. In TriR, we jointly consider the visual, linguistic, and stylistic alignments to generate factual, fluent, and natural caption in the target language. To be specific, the image-source caption annotation provides factual semantic guidance, whereas the unrelated target corpus guides the language style of generated caption. To achieve this, we construct a visual reward module to measure the cross-modal semantic embedding of image and target caption, a linguistic reward module to measure the cross-linguistic embedding of source and target captions, and a stylistic reward module to imitate the presentation style of target corpus. The TriR can be implemented with either classical CNN-LSTM or prevalent Transformer architecture. Extensive experiments are conducted with four cross-lingual settings, i.e., Chinese-to-English, English-to-Chinese, English-to-German and English-to-French. Experimental results demonstrate the remarkable superiority of our method, and sufficient ablation experiments validate the beneficial impact of every reward.
- 11 January 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 20 (4) , 1-23
Open AccessVisual-linguistic-stylistic Triple Reward for Cross-lingual Image Captioning
Generating image captions in different languages is worth exploring and essential for non-native speakers. Nevertheless, collecting paired annotation for every language is time-consuming and impractical, particularly for minor languages. To this end, the cross-lingual image captioning task is proposed, which leverages existing image-source caption annotation data and wild unrelated target corpus to generate satisfactory caption in the target language. Current methods perform a two-step translation process of image-to-pivot (source) and pivot-to-target. The distinct two-step process comes with certain caption issues, such as the weak semantic alignment between the image and the generated caption and the generated caption’s non-target language style. To address these issues, we propose an end-to-end reinforce learning framework with Visual-Linguistic-Stylistic Triple Reward named TriR. In TriR, we jointly consider the visual, linguistic, and stylistic alignments to generate factual, fluent, and natural caption in the target language. To be specific, the image-source caption annotation provides factual semantic guidance, whereas the unrelated target corpus guides the language style of generated caption. To achieve this, we construct a visual reward module to measure the cross-modal semantic embedding of image and target caption, a linguistic reward module to measure the cross-linguistic embedding of source and target captions, and a stylistic reward module to imitate the presentation style of target corpus. The TriR can be implemented with either classical CNN-LSTM or prevalent Transformer architecture. Extensive experiments are conducted with four cross-lingual settings, i.e., Chinese-to-English, English-to-Chinese, English-to-German and English-to-French. Experimental results demonstrate the remarkable superiority of our method, and sufficient ablation experiments validate the beneficial impact of every reward.