PQPenggang Qin

Penggang Qin

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Penggang Qin has a total of 10 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Scenario-Aware Multimodal Chain-of-Thought Prompting for Rationales of VideoSocial Relations

    by Penggang Qin, Tong Xu, Chao Zhang, Heda Wang, Yao Hu, Enhong Chen

    Recently, audiences have increasingly watched diverse dramas and movies on streaming platforms, prompting platform administrators to enhance their understanding of video semantics. In particular, capturing and interpreting social relations among characters is critical for content-driven intelligent services and enhancing user experience. However, most existing research has solely approached social relations recognition as a classification problem, without justifiable interpretation of prediction results within the video context. To address this issue, we study cognitive science research and the Chain-of-Thought (CoT) strategy. Based on these foundations, we propose SaMo-CoT, an approach that leverages large language models (LLMs) in step-by-step reasoning. This approach simulates human cognitive processes by combining social scenarios in videos and incorporating empirical social interaction knowledge. In this way, we enable verbal rationales for determining social relations in video understanding. Furthermore, we present an innovative doubly-right social relation recognition framework that predicts both correct social relation labels and correct scenario rationales. Specifically, we translate verbal CoT into multimodal CoT by leveraging scenario-aware prompts and contrastive learning. Extensive experiments demonstrate significant improvements in classification accuracy and interpretability compared to traditional approaches.


  • Open Access

    InteractNet: Social Interaction Recognition for Semantic-rich Videos

    by Yuanjie Lyu, Penggang Qin, Tong Xu, Chen Zhu, Enhong Chen

    The overwhelming surge of online video platforms has raised an urgent need for social interaction recognition techniques. Compared with simple short-term actions, long-term social interactions in semantic-rich videos could reflect more complicated semantics such as character relationships or emotions, which will better support various downstream applications, e.g., story summarization and fine-grained clip retrieval. However, considering the longer duration of social interactions with severe mutual overlap, involving multiple characters, dynamic scenes, and multi-modal cues, among other factors, traditional solutions for short-term action recognition may probably fail in this task. To address these challenges, in this article, we propose a hierarchical graph-based system, named InteractNet, to recognize social interactions in a multi-modal perspective. Specifically, our approach first generates a semantic graph for each sampled frame with integrating multi-modal cues and then learns the node representations as short-term interaction patterns via an adapted GCN module. Along this line, global interaction representations are accumulated through a sub-clip identification module, effectively filtering out irrelevant information and resolving temporal overlaps between interactions. In the end, the association among simultaneous interactions will be captured and modelled by constructing a global-level character-pair graph to predict the final social interactions. Comprehensive experiments on publicly available datasets demonstrate the effectiveness of our approach compared with state-of-the-art baseline methods.


  • When I Fall in Love: Capturing Video-Oriented Social Relationship Evolution via Attentive GNN

    by Penggang Qin, Shiwei Wu, Tong Xu, Yanbin Hao, Fuli Feng, Chen Zhu, Enhong Chen

    With the booming of streaming media platforms, viewers now get used to watching dramas and movies via online platforms with more intelligent services. Usually, character relationships may dynamically evolve with stories promoting in long videos. Therefore, automatic tools to capture the social relation evolution among characters are urgently required to enrich the viewing experience. However, most existing works mainly focus on shorter isolated video clips. Considering the development of the plot, they may fail to effectively summarize relationships as holistic semantic representations for the whole video. To deal with these challenges, in this paper, we propose a novel Dynamic-Evolutionary Graph Attention Network (DE-GAT) framework to generate the evolving social relation graph among characters and capture the characters’ relation evolutionary trajectory throughout the entire video. DE-GAT first integrates the multimodal cues, including visual and textual information in each video clip via the graph attention network (GAT). Expanding the temporal receptive field from clip-level to scenario-level, the most relevant factors of the evolution of social relationships can be explored. Eventually, all the scenario-level social graphs are merged to obtain the evolving global social graph for the entire movie. Extensive evaluations on the real-world MovieGraphs dataset have validated the positive impact of temporal receptive field expansion and multimodal cues on capturing evolving social relations.