MIKO: Multimodal Intention Knowledge Distillation from Large Language Models for Social-Media Commonsense Discovery
Preprint
Abstract
Social media has become a ubiquitous tool for connecting with others, staying updated with news, expressing opinions, and finding entertainment. However, understanding the intention behind social media posts remains challenging due to the implicitness of intentions in social media posts, the need for cross-modality understanding of both text and images, and the presence of noisy information such as hashtags, misspelled words, and complicated abbreviations. To address these challenges, we present MIKO, a Multimodal Intention Kowledge DistillatiOn framework that collaboratively leverages a Large Language Model (LLM) and a Multimodal Large Language Model (MLLM) to uncover users' intentions. Specifically, we use an MLLM to interpret the image and an LLM to extract key information from the text and finally instruct the LLM again to generate intentions. By applying MIKO to publicly available social media datasets, we construct an intention knowledge base featuring 1,372K intentions rooted in 137,287 posts. We conduct a two-stage annotation to verify the quality of the generated knowledge and benchmark the performance of widely used LLMs for intention generation. We further apply MIKO to a sarcasm detection dataset and distill a student model to demonstrate the downstream benefits of applying intention knowledge.Keywords
All Related Versions
This publication has 31 references indexed in Scilit:
- Improved Baselines with Visual Instruction Tuning2024
- Domain Knowledge Distillation from Large Language Model: An Empirical Study in the Autonomous Driving Domain2023
- Exploring information avoidance intention of social media users: a cognition–affect–conation perspectiveInternet Research, 2020
- User sentiment analysis based on social network information and its application in consumer reconstruction intentionComputers in Human Behavior, 2019
- Context and Short Term User Intention Aware Hybrid Session Based Recommendation System2019
- Hate Speech on Twitter: A Pragmatic Approach to Collect Hateful and Offensive Expressions and Perform Hate Speech DetectionIEEE Access, 2018
- Twitter100k: A Real-World Dataset for Weakly Supervised Cross-Media RetrievalIEEE Transactions on Multimedia, 2017
- Linking dimensions of social media use to job performance: The role of social capitalThe Journal of Strategic Information Systems, 2015
- What Happens When Recommendation System Meets Reputation System? The Impact of Recommendation Information on Purchase Intention2013
- Framewise phoneme classification with bidirectional LSTM and other neural network architecturesNeural Networks, 2005