Miko: Multimodal Intention Knowledge Distillation from Large Language Models for Social-Media Commonsense Discovery
Open Access
- 28 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 3303-3312
Abstract
Social media has become ubiquitous for connecting with others, staying updated with news, expressing opinions, and finding entertainment. However, understanding the intention behind social media posts remains challenging due to the implicit and commonsense nature of these intentions, the need for cross-modality understanding of both text and images, and the presence of noisy information such as hashtags, misspelled words, and complicated abbreviations. To address these challenges, we present MIKO, a Multimodal Intention Knowledge DistillatiOn framework that collaboratively leverages a Large Language Model (LLM) and a Multimodal Large Language Model (MLLM) to uncover users' intentions. Specifically, our approach uses an MLLM to interpret the image, an LLM to extract key information from the text, and another LLM to generate intentions. By applying MIKO to publicly available social media datasets, we construct an intention knowledge base featuring 1,372K intentions rooted in 137,287 posts. Moreover, We conduct a two-stage annotation to verify the quality of the generated knowledge and benchmark the performance of widely used LLMs for intention generation, and further apply MIKO to a sarcasm detection dataset and distill a student model to demonstrate the downstream benefits of applying intention knowledge.Keywords
All Related Versions
Funding Information
- NSFC (U20B2053, 62302023)
- RIF (R6020-19, R6021-20)
- GRF (16211520, 16205322)
- UGC Research Matching Grants (RMGS20EG01-D, RMGS20CR11, RMGS20CR12, RMGS20EG19, RMGS20EG21, RMGS23CR05, RMGS23EG08)
This publication has 27 references indexed in Scilit:
- Acquiring and modeling abstract commonsense knowledge via conceptualizationArtificial Intelligence, 2024
- Room-and-Object Aware Knowledge Reasoning for Remote Embodied Referring Expression2021
- Exploring information avoidance intention of social media users: a cognition–affect–conation perspectiveInternet Research, 2020
- User sentiment analysis based on social network information and its application in consumer reconstruction intentionComputers in Human Behavior, 2019
- Context and Short Term User Intention Aware Hybrid Session Based Recommendation System2019
- Hate Speech on Twitter: A Pragmatic Approach to Collect Hateful and Offensive Expressions and Perform Hate Speech DetectionIEEE Access, 2018
- Twitter100k: A Real-World Dataset for Weakly Supervised Cross-Media RetrievalIEEE Transactions on Multimedia, 2017
- Linking dimensions of social media use to job performance: The role of social capitalThe Journal of Strategic Information Systems, 2015
- What Happens When Recommendation System Meets Reputation System? The Impact of Recommendation Information on Purchase Intention2013
- Framewise phoneme classification with bidirectional LSTM and other neural network architecturesNeural Networks, 2005