Shuo Li
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Shuo Li has a total of 5 co-authors. Here are the top co-authors by number of shared publications:
- Linmei Hu
Beijing Institute of Technology, Beijing, China
1 shared publication - Liqiang Jing
University of Texas at Dallas
1 shared publication - Liqiang Nie
Harbin Institute of Technology (Shenzhen), Shenzhen, China
1 shared publication - Xiaolin Chen
School of Software, Joint SDU-NTU Centre for Artificial Intelligence Research, Shandong University, Jinan, China
1 shared publication - Xuemeng Song
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China
1 shared publication
Latest Publications
- Open Access
- 7 November 2023
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Information Systems
- Vol. 42 (2) , 1-25
Multimodal Dialog Systems with Dual Knowledge-enhanced Generative Pretrained Language Model
Text response generation for multimodal task-oriented dialog systems, which aims to generate the proper text response given the multimodal context, is an essential yet challenging task. Although existing efforts have achieved compelling success, they still suffer from two pivotal limitations: (1) overlook the benefit of generative pretraining and (2) ignore the textual context-related knowledge . To address these limitations, we propose a novel dual knowledge-enhanced generative pretrained language mode for multimodal task-oriented dialog systems (DKMD), consisting of three key components: dual knowledge selection , dual knowledge-enhanced context learning , and knowledge-enhanced response generation . To be specific, the dual knowledge selection component aims to select the related knowledge according to both textual and visual modalities of the given context. Thereafter, the dual knowledge-enhanced context learning component targets seamlessly, integrating the selected knowledge into the multimodal context learning from both global and local perspectives, where the cross-modal semantic relation is also explored. Moreover, the knowledge-enhanced response generation component comprises a revised BART decoder, where an additional dot-product knowledge-decoder attention sub-layer is introduced for explicitly utilizing the knowledge to advance the text response generation. Extensive experiments on a public dataset verify the superiority of the proposed DKMD over state-of-the-art competitors.
- 7 November 2023
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Information Systems
- Vol. 42 (2) , 1-25
Open AccessMultimodal Dialog Systems with Dual Knowledge-enhanced Generative Pretrained Language Model
Text response generation for multimodal task-oriented dialog systems, which aims to generate the proper text response given the multimodal context, is an essential yet challenging task. Although existing efforts have achieved compelling success, they still suffer from two pivotal limitations: (1) overlook the benefit of generative pretraining and (2) ignore the textual context-related knowledge . To address these limitations, we propose a novel dual knowledge-enhanced generative pretrained language mode for multimodal task-oriented dialog systems (DKMD), consisting of three key components: dual knowledge selection , dual knowledge-enhanced context learning , and knowledge-enhanced response generation . To be specific, the dual knowledge selection component aims to select the related knowledge according to both textual and visual modalities of the given context. Thereafter, the dual knowledge-enhanced context learning component targets seamlessly, integrating the selected knowledge into the multimodal context learning from both global and local perspectives, where the cross-modal semantic relation is also explored. Moreover, the knowledge-enhanced response generation component comprises a revised BART decoder, where an additional dot-product knowledge-decoder attention sub-layer is introduced for explicitly utilizing the knowledge to advance the text response generation. Extensive experiments on a public dataset verify the superiority of the proposed DKMD over state-of-the-art competitors.