Zheng Wang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Zheng Wang has a total of 53 co-authors. Here are the top co-authors by number of shared publications:
- Cheng Long
Nanyang technological university, Singapore, Singapore
6 shared publications - Wei Shi
Huawei Singapore Research Center, Singapore, Singapore
5 shared publications - Gao Cong
Nanyang Technological University
4 shared publications - Shihao Xu
Huawei Singapore Research Center, Singapore, Singapore
3 shared publications - Tat-Seng Chua
National University of Singapore, Singapore, Singapore
3 shared publications - Bingzheng Gan
Huawei Singapore Research Center, Singapore, Singapore
2 shared publications
Latest Publications
- Open Access
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 14320-14322
Large Generative Models Meet Multimodal Applications (LGM 3 A)
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 14320-14322
Open AccessLarge Generative Models Meet Multimodal Applications (LGM 3 A)
- Open Access
- 26 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 3-10
PromptBoost: Annotation-Free Prompt Expansion for Chinese Text-to-Image Generation
- 26 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 3-10
Open AccessPromptBoost: Annotation-Free Prompt Expansion for Chinese Text-to-Image Generation
- 19 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 36 (2) , 1810-1822
TrajSV: A Trajectory-Based Model for Sports Video Representations and Applications
Sports analytics has received significant attention from both academia and industry in recent years. Despite the growing interest and efforts in this field, several issues remain unresolved, including (1) data unavailability, (2) lack of an effective trajectory-based framework, and (3) requirement for sufficient supervision labels. In this paper, we present TrajSV, a trajectory-based framework that addresses various issues in existing studies. TrajSV comprises three components: data preprocessing, Clip Representation Network (CRNet), and Video Representation Network (VRNet). The data preprocessing module extracts player and ball trajectories from sports broadcast videos. CRNet utilizes a trajectory-enhanced Transformer module to learn clip representations based on these trajectories. Additionally, VRNet learns video representations by aggregating clip representations and visual features with an encoder-decoder architecture. Finally, a triple contrastive loss is introduced to optimize both video and clip representations in an unsupervised manner. The experiments are conducted on three broadcast video datasets to verify the effectiveness of TrajSV for three types of sports (i.e., soccer, basketball, and volleyball) with three downstream applications (i.e., sports video retrieval, action spotting, and video captioning). The results demonstrate that TrajSV achieves state-of-the-art performance in sports video retrieval, showcasing a nearly 70% improvement. It outperforms baselines in action spotting, achieving state-of-the-art results in 9 out of 17 action categories, and demonstrates a nearly 20% improvement in video captioning. Additionally, we introduce a deployed system along with the three applications based on TrajSV.
- 19 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 36 (2) , 1810-1822
TrajSV: A Trajectory-Based Model for Sports Video Representations and Applications
Sports analytics has received significant attention from both academia and industry in recent years. Despite the growing interest and efforts in this field, several issues remain unresolved, including (1) data unavailability, (2) lack of an effective trajectory-based framework, and (3) requirement for sufficient supervision labels. In this paper, we present TrajSV, a trajectory-based framework that addresses various issues in existing studies. TrajSV comprises three components: data preprocessing, Clip Representation Network (CRNet), and Video Representation Network (VRNet). The data preprocessing module extracts player and ball trajectories from sports broadcast videos. CRNet utilizes a trajectory-enhanced Transformer module to learn clip representations based on these trajectories. Additionally, VRNet learns video representations by aggregating clip representations and visual features with an encoder-decoder architecture. Finally, a triple contrastive loss is introduced to optimize both video and clip representations in an unsupervised manner. The experiments are conducted on three broadcast video datasets to verify the effectiveness of TrajSV for three types of sports (i.e., soccer, basketball, and volleyball) with three downstream applications (i.e., sports video retrieval, action spotting, and video captioning). The results demonstrate that TrajSV achieves state-of-the-art performance in sports video retrieval, showcasing a nearly 70% improvement. It outperforms baselines in action spotting, achieving state-of-the-art results in 9 out of 17 action categories, and demonstrates a nearly 20% improvement in video captioning. Additionally, we introduce a deployed system along with the three applications based on TrajSV.
- Open Access
- 13 July 2025
- Published by Association for Computing Machinery (ACM)
- p. 1413-1422
InstructRAG: Leveraging Retrieval-Augmented Generation on Instruction Graphs for LLM-Based Task Planning
- 13 July 2025
- Published by Association for Computing Machinery (ACM)
- p. 1413-1422
Open AccessInstructRAG: Leveraging Retrieval-Augmented Generation on Instruction Graphs for LLM-Based Task Planning
- Open Access
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 36-44
A Method for Efficient Structured Data Generation with Large Language Models
With the rapid development of large language model technology, we find ourselves at an interesting juncture regarding the importance of data. The textual data samples from these large unsupervised models are often of poor quality, which in turn produces substandard results. Implicitly, this means that the model struggles to learn the exact underlying structure of the data distribution without supervision, which can manifest as output lacking fidelity and relevance to real data distributions. In order to overcome some of these limitations in data-driven text generation tasks, this paper presents a Efficient Data Generation System (EDGS) for multimodal structured data generation. EDGS achieves this by using clustering abstraction to process various data input types through templates, thereby enabling quick data generation and reducing resource consumption. EDGS is robust, ensuring stable and reliable performance across different conditions and is more cost-effective in terms of token usage. Traditional methods handle data on a per-instance basis without using templates, but for EDGS, by abstracting and clustering different input types, it efficiently generates data from large models. Its adaptability makes it ideal for multi-modal data generation tasks since it relies on the fundamental functions of general large-scale language models. A query-answer bidirectional generation mechanism is used to rapidly amplify data with the system's help.
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 36-44
Open AccessA Method for Efficient Structured Data Generation with Large Language Models
With the rapid development of large language model technology, we find ourselves at an interesting juncture regarding the importance of data. The textual data samples from these large unsupervised models are often of poor quality, which in turn produces substandard results. Implicitly, this means that the model struggles to learn the exact underlying structure of the data distribution without supervision, which can manifest as output lacking fidelity and relevance to real data distributions. In order to overcome some of these limitations in data-driven text generation tasks, this paper presents a Efficient Data Generation System (EDGS) for multimodal structured data generation. EDGS achieves this by using clustering abstraction to process various data input types through templates, thereby enabling quick data generation and reducing resource consumption. EDGS is robust, ensuring stable and reliable performance across different conditions and is more cost-effective in terms of token usage. Traditional methods handle data on a per-instance basis without using templates, but for EDGS, by abstracting and clustering different input types, it efficiently generates data from large models. Its adaptability makes it ideal for multi-modal data generation tasks since it relies on the fundamental functions of general large-scale language models. A query-answer bidirectional generation mechanism is used to rapidly amplify data with the system's help.
- Open Access
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-3
LGM 3 A '24: the 2nd Workshop on Large Generative Models Meet Multimodal Applications
This workshop aims to explore the potential of large generative models to revolutionize how we interact with multimodal information. A Large Language Model (LLM) represents a sophisticated form of artificial intelligence engineered to comprehend and produce natural language text, exemplified by technologies such as GPT, LLaMA, Flan-T5, ChatGLM, Qwen, etc. These models undergo training on extensive text datasets, exhibiting commendable attributes including robust language generation, zero-shot transfer capabilities, and In-Context Learning (ICL). With the surge in multimodal content-encompassing images, videos, audio, and 3D models-over the recent period, Large MultiModal Models (LMMs) have seen significant enhancements. These improvements enable the augmentation of conventional LLMs to accommodate multimodal inputs or outputs, as seen in BLIP, Flamingo, KOSMOS, LLaVA, Gemini, GPT-4, etc. Concurrently, certain research initiatives have developed specific modalities, with Kosmos2 and MiniGPT-5 focusing on image generation, and SpeechGPT on speech production. There are also endeavors to integrate LLMs with external tools to achieve a near "any-to-any" multimodal comprehension and generation capacity, illustrated by projects like Visual-ChatGPT, ViperGPT, MMREACT, HuggingGPT, and AudioGPT. Collectively,these models, spanning not only text and image generation but also other modalities, are referred to as large generative models. This workshop will allow researchers, practitioners, and industry professionals to explore the latest trends and best practices in the multimodal applications of large generative models.
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Open AccessLGM 3 A '24: the 2nd Workshop on Large Generative Models Meet Multimodal Applications
This workshop aims to explore the potential of large generative models to revolutionize how we interact with multimodal information. A Large Language Model (LLM) represents a sophisticated form of artificial intelligence engineered to comprehend and produce natural language text, exemplified by technologies such as GPT, LLaMA, Flan-T5, ChatGLM, Qwen, etc. These models undergo training on extensive text datasets, exhibiting commendable attributes including robust language generation, zero-shot transfer capabilities, and In-Context Learning (ICL). With the surge in multimodal content-encompassing images, videos, audio, and 3D models-over the recent period, Large MultiModal Models (LMMs) have seen significant enhancements. These improvements enable the augmentation of conventional LLMs to accommodate multimodal inputs or outputs, as seen in BLIP, Flamingo, KOSMOS, LLaVA, Gemini, GPT-4, etc. Concurrently, certain research initiatives have developed specific modalities, with Kosmos2 and MiniGPT-5 focusing on image generation, and SpeechGPT on speech production. There are also endeavors to integrate LLMs with external tools to achieve a near "any-to-any" multimodal comprehension and generation capacity, illustrated by projects like Visual-ChatGPT, ViperGPT, MMREACT, HuggingGPT, and AudioGPT. Collectively,these models, spanning not only text and image generation but also other modalities, are referred to as large generative models. This workshop will allow researchers, practitioners, and industry professionals to explore the latest trends and best practices in the multimodal applications of large generative models.
- Open Access
- 14 October 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Knowledge Discovery From Data
- Vol. 18 (9) , 1-27
Billiards Sports Analytics: Datasets and Tasks
Nowadays, it becomes a common practice to capture some data of sports games with devices such as GPS sensors and cameras and then use the data to perform various analyses on sports games, including tactics discovery, similar game retrieval, performance study, and so forth. While this practice has been conducted to many sports such as basketball and soccer, it remains largely unexplored on the billiards sports, which is mainly due to the lack of publicly available datasets. Motivated by this, we collect a dataset of billiards sports, which includes the layouts (i.e., locations) of billiards balls after performing break shots, called break shot layouts, the traces of the balls as a result of strikes (in the form of trajectories), and detailed statistics and performance indicators. We then study and develop techniques for three tasks on the collected dataset, including (1) prediction and (2) generation on the layouts data, and (3) similar billiards layout retrieval on the layouts data, which can serve different users such as coaches, players and fans. We conduct extensive experiments on the collected dataset and the results show that our methods perform effectively and efficiently.
- 14 October 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Knowledge Discovery From Data
- Vol. 18 (9) , 1-27
Open AccessBilliards Sports Analytics: Datasets and Tasks
Nowadays, it becomes a common practice to capture some data of sports games with devices such as GPS sensors and cameras and then use the data to perform various analyses on sports games, including tactics discovery, similar game retrieval, performance study, and so forth. While this practice has been conducted to many sports such as basketball and soccer, it remains largely unexplored on the billiards sports, which is mainly due to the lack of publicly available datasets. Motivated by this, we collect a dataset of billiards sports, which includes the layouts (i.e., locations) of billiards balls after performing break shots, called break shot layouts, the traces of the balls as a result of strikes (in the form of trajectories), and detailed statistics and performance indicators. We then study and develop techniques for three tasks on the collected dataset, including (1) prediction and (2) generation on the layouts data, and (3) similar billiards layout retrieval on the layouts data, which can serve different users such as coaches, players and fans. We conduct extensive experiments on the collected dataset and the results show that our methods perform effectively and efficiently.
- Open Access
- 13 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 1374-1385
Multimodal Query Suggestion with Multi-Agent Reinforcement Learning from Human Feedback
In the rapidly evolving landscape of information retrieval, search engines strive to provide more personalized and relevant results to users. Query suggestion systems play a crucial role in achieving this goal by assisting users in formulating effective queries. However, existing query suggestion systems mainly rely on textual inputs, potentially limiting user search experiences for querying images. In this paper, we introduce a novel Multimodal Query Suggestion (MMQS) task, which aims to generate query suggestions based on user query images to improve the intentionality and diversity of search results. We present the RL4Sugg framework, leveraging the power of Large Language Models (LLMs) with Multi-Agent Reinforcement Learning from Human Feedback to optimize the generation process. Through comprehensive experiments, we validate the effectiveness of RL4Sugg, demonstrating a 18% improvement compared to the best existing approach. Moreover, the MMQS has been transferred into real-world search engine products, which yield enhanced user engagement. Our research advances query suggestion systems and provides a new perspective on multimodal information retrieval.
- 13 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 1374-1385
Open AccessMultimodal Query Suggestion with Multi-Agent Reinforcement Learning from Human Feedback
In the rapidly evolving landscape of information retrieval, search engines strive to provide more personalized and relevant results to users. Query suggestion systems play a crucial role in achieving this goal by assisting users in formulating effective queries. However, existing query suggestion systems mainly rely on textual inputs, potentially limiting user search experiences for querying images. In this paper, we introduce a novel Multimodal Query Suggestion (MMQS) task, which aims to generate query suggestions based on user query images to improve the intentionality and diversity of search results. We present the RL4Sugg framework, leveraging the power of Large Language Models (LLMs) with Multi-Agent Reinforcement Learning from Human Feedback to optimize the generation process. Through comprehensive experiments, we validate the effectiveness of RL4Sugg, demonstrating a 18% improvement compared to the best existing approach. Moreover, the MMQS has been transferred into real-world search engine products, which yield enhanced user engagement. Our research advances query suggestion systems and provides a new perspective on multimodal information retrieval.
- Open Access
- 29 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 7-14
NeurSEG: A Segment Driven Deep Neural Model for Nested Named Entity Recognition
Named Entity Recognition (NER) is a fundamental problem in natural language processing (NLP). Apart from flat entities, nested entities are also commonly existed in real-life textual data. However, the current methods are not capable of handling nested structures for NER effectively. In this paper, we propose a novel segment driven modeling method (NeurSEG) for the nested NER problem, which can effectively extract entities from the nested structures in complex nesting scenarios. The proposed NeurSEG model first finds the nested label of each word in a sentence and determines the positional relationships between neighbouring words, and then extracts the entities and predicts the corresponding entity types. In addition, we also propose an augmented training method for further improving the performance. We have conducted extensive experiments based on both flat and nested NER benchmark datasets. The performance results have shown that our proposed NeurSEG model has achieved promising performance while retaining its runtime efficiency for the nested NER task. Moreover, the proposed model has also achieved very competitive results when compared with the existing models for the flat NER task, demonstrating its capability for tackling both nested and flat NER tasks.
- 29 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 7-14
Open AccessNeurSEG: A Segment Driven Deep Neural Model for Nested Named Entity Recognition
Named Entity Recognition (NER) is a fundamental problem in natural language processing (NLP). Apart from flat entities, nested entities are also commonly existed in real-life textual data. However, the current methods are not capable of handling nested structures for NER effectively. In this paper, we propose a novel segment driven modeling method (NeurSEG) for the nested NER problem, which can effectively extract entities from the nested structures in complex nesting scenarios. The proposed NeurSEG model first finds the nested label of each word in a sentence and determines the positional relationships between neighbouring words, and then extracts the entities and predicts the corresponding entity types. In addition, we also propose an augmented training method for further improving the performance. We have conducted extensive experiments based on both flat and nested NER benchmark datasets. The performance results have shown that our proposed NeurSEG model has achieved promising performance while retaining its runtime efficiency for the nested NER task. Moreover, the proposed model has also achieved very competitive results when compared with the existing models for the flat NER task, demonstrating its capability for tackling both nested and flat NER tasks.