Yidan Sun
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Yidan Sun has a total of 4 co-authors. Here are the top co-authors by number of shared publications:
- Boyang Li
College of Computing and Data Science, Nanyang Technological University, Singapore, Singapore
1 shared publication - Xu Guo
Nanyang technological university, Singapore, Singapore
1 shared publication - Yunxin Li
Department of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China
1 shared publication - Zilin Du
Nanyang technological university, Singapore, Singapore
1 shared publication
Latest Publications
- Open Access
- 26 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 5504-5513
Training Multimedia Event Extraction With Generated Images and Captions
Contemporary news reporting increasingly features multimedia content, motivating research on multimedia event extraction. However, the task lacks annotated multimodal training data and artificially generated training data suffer from the distribution shift from the real-world data. In this paper, we propose Cross-modality Augmented Multimedia Event Learning (CAMEL), which successfully utilizes artificially generated multimodal training data and achieves state-of-the-art performance. Conditioned on unimodal training data, we generate multimodal training data using off-the-shelf image generators like Stable Diffusion [45] and image captioners like BLIP [24]. After that, we train the network on the resultant multimodal datasets. In order to learn robust features that are effective across domains, we devise an iterative and gradual training strategy. Substantial experiments show that CAMEL surpasses state-of-the-art (SOTA) baselines on the M2E2 benchmark. On multimedia events in particular, we outperform the prior SOTA by 4.2% F1 on event mention identification and by 9.8% F1 on argument identification, which demonstrates that CAMEL learns synergistic representations from the two modalities. Our work demonstrates a recipe to unleash the power of synthetic training data in structured prediction.
- 26 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 5504-5513
Open AccessTraining Multimedia Event Extraction With Generated Images and Captions
Contemporary news reporting increasingly features multimedia content, motivating research on multimedia event extraction. However, the task lacks annotated multimodal training data and artificially generated training data suffer from the distribution shift from the real-world data. In this paper, we propose Cross-modality Augmented Multimedia Event Learning (CAMEL), which successfully utilizes artificially generated multimodal training data and achieves state-of-the-art performance. Conditioned on unimodal training data, we generate multimodal training data using off-the-shelf image generators like Stable Diffusion [45] and image captioners like BLIP [24]. After that, we train the network on the resultant multimodal datasets. In order to learn robust features that are effective across domains, we devise an iterative and gradual training strategy. Substantial experiments show that CAMEL surpasses state-of-the-art (SOTA) baselines on the M2E2 benchmark. On multimedia events in particular, we outperform the prior SOTA by 4.2% F1 on event mention identification and by 9.8% F1 on argument identification, which demonstrates that CAMEL learns synergistic representations from the two modalities. Our work demonstrates a recipe to unleash the power of synthetic training data in structured prediction.