Jiaming Lei
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Jiaming Lei has a total of 9 co-authors. Here are the top co-authors by number of shared publications:
- Lin Li
AI Chip Center for Emerging Smart Systems, Hong Kong, China and The Hong Kong University of Science and Technology, Hong Kong, China
2 shared publications - Chunping Wang
FinVolution Group, Shanghai, China
1 shared publication Jun Xiao
1 shared publication- Jun Xiao
Zhejiang University, Hangzhou, Zhejiang, China
1 shared publication - Lei Chen
FinVolution Group, Shanghai, China
1 shared publication Long Chen
1 shared publication
Latest Publications
- 5 May 2025
- Published by Elsevier in Pattern Recognition
- Vol. 167
Knowledge Integration for Grounded Situation Recognition
- 5 May 2025
- Published by Elsevier in Pattern Recognition
- Vol. 167
Knowledge Integration for Grounded Situation Recognition
- Open Access
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 1602-1611
Seeing Beyond Classes: Zero-Shot Grounded Situation Recognition via Language Explainer
Benefiting from strong generalization ability, pre-trained vision-language models (VLMs), e.g., CLIP, have been widely utilized in zero-shot scene understanding. Unlike simple recognition tasks, grounded situation recognition (GSR) requires the model not only to classify salient activity (verb) in the image, but also to detect all semantic roles that participate in the action. This complex task usually involves three steps: verb recognition, semantic role grounding, and noun recognition. Directly employing class-based prompts with VLMs and grounding models for this task suffers from several limitations, e.g., it struggles to distinguish ambiguous verb concepts, accurately localize roles with fixed verb-centric template input, and achieve context-aware noun predictions. In this paper, we argue that these limitations stem from the model's poor understanding of verb/noun classes. To this end, we introduce a new approach for zero-shot GSR via Language EXplainer (LEX), which significantly boosts the model's comprehensive capabilities through three explainers: 1) verb explainer, which generates general verb-centric descriptions to enhance the discriminability of different verb classes; 2) grounding explainer, which rephrases verb-centric templates for clearer understanding, thereby enhancing precise semantic role localization; and 3) noun explainer, which creates scene-specific noun descriptions to ensure context-aware noun recognition. By equipping each step of the GSR process with an auxiliary explainer, LEX facilitates complex scene understanding in real-world scenarios. Our extensive validations on the SWiG dataset demonstrate LEX's effectiveness and interoperability in zero-shot GSR.
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 1602-1611
Open AccessSeeing Beyond Classes: Zero-Shot Grounded Situation Recognition via Language Explainer
Benefiting from strong generalization ability, pre-trained vision-language models (VLMs), e.g., CLIP, have been widely utilized in zero-shot scene understanding. Unlike simple recognition tasks, grounded situation recognition (GSR) requires the model not only to classify salient activity (verb) in the image, but also to detect all semantic roles that participate in the action. This complex task usually involves three steps: verb recognition, semantic role grounding, and noun recognition. Directly employing class-based prompts with VLMs and grounding models for this task suffers from several limitations, e.g., it struggles to distinguish ambiguous verb concepts, accurately localize roles with fixed verb-centric template input, and achieve context-aware noun predictions. In this paper, we argue that these limitations stem from the model's poor understanding of verb/noun classes. To this end, we introduce a new approach for zero-shot GSR via Language EXplainer (LEX), which significantly boosts the model's comprehensive capabilities through three explainers: 1) verb explainer, which generates general verb-centric descriptions to enhance the discriminability of different verb classes; 2) grounding explainer, which rephrases verb-centric templates for clearer understanding, thereby enhancing precise semantic role localization; and 3) noun explainer, which creates scene-specific noun descriptions to ensure context-aware noun recognition. By equipping each step of the GSR process with an auxiliary explainer, LEX facilitates complex scene understanding in real-world scenarios. Our extensive validations on the SWiG dataset demonstrate LEX's effectiveness and interoperability in zero-shot GSR.