JZJing Zhang

Jing Zhang

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Jing Zhang has a total of 7 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    EmoSEM: Segment and Explain Emotion Stimuli in Visual Art

    by Jing Zhang, Dan Guo, Zhangbin Li, Meng Wang

  • Open Access

    Syntax-Controllable Video Captioning with Tree-Structural Syntax Augmentation

    by Jiahui Sun, Peipei Song, Jing Zhang, Dan Guo

    Traditional video captioning tasks often suffer from uncontrollable syntax, limiting their adaptability to diverse practical requirements. This paper explores a challenging task known as syntax-controllable video captioning (SCVC). Specifically, this task aims to generate captions for videos that convey the visual content accurately and adhere to the syntactic structure provided in an example sentence. To tackle the SCVC task, we propose a novel Tree-Structural Syntax Augmentation Network (TSAN). It consists of a tree-structural syntax encoder to capture syntactic parent-child relationships of example sentences and a syntax-customized caption generator to integrate syntax and semantic hints effectively. Moreover, we introduce a syntax-semantic joint optimization strategy that considers both syntax and semantic constraints for joint training. Considering the fact that there are no available syntax-controllable video caption datasets, we use the traditional video captioning dataset to train the model. In the inference process, the model generates syntax-controllable captions in a zero-shot manner, applying exemplar sentences randomly selected from an open corpus and unseen in the training data. The experiments on MSRVTT and VATEX datasets validate the superior performance of TSAN in both semantic and syntactic evaluations. Extensive ablation studies and visualizations further show the effectiveness of the proposed method.


  • Open Access

    Visual-linguistic-stylistic Triple Reward for Cross-lingual Image Captioning

    by Jing Zhang, Dan Guo, Xun Yang, Peipei Song, Meng Wang

    Generating image captions in different languages is worth exploring and essential for non-native speakers. Nevertheless, collecting paired annotation for every language is time-consuming and impractical, particularly for minor languages. To this end, the cross-lingual image captioning task is proposed, which leverages existing image-source caption annotation data and wild unrelated target corpus to generate satisfactory caption in the target language. Current methods perform a two-step translation process of image-to-pivot (source) and pivot-to-target. The distinct two-step process comes with certain caption issues, such as the weak semantic alignment between the image and the generated caption and the generated caption’s non-target language style. To address these issues, we propose an end-to-end reinforce learning framework with Visual-Linguistic-Stylistic Triple Reward named TriR. In TriR, we jointly consider the visual, linguistic, and stylistic alignments to generate factual, fluent, and natural caption in the target language. To be specific, the image-source caption annotation provides factual semantic guidance, whereas the unrelated target corpus guides the language style of generated caption. To achieve this, we construct a visual reward module to measure the cross-modal semantic embedding of image and target caption, a linguistic reward module to measure the cross-linguistic embedding of source and target captions, and a stylistic reward module to imitate the presentation style of target corpus. The TriR can be implemented with either classical CNN-LSTM or prevalent Transformer architecture. Extensive experiments are conducted with four cross-lingual settings, i.e., Chinese-to-English, English-to-Chinese, English-to-German and English-to-French. Experimental results demonstrate the remarkable superiority of our method, and sufficient ablation experiments validate the beneficial impact of every reward.