Meme Trojan: Backdoor Attacks Against Hateful Meme Detection via Cross-Modal Triggers
Open Access
- 11 April 2025
- journal article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (8) , 7844-7852
Abstract
Hateful meme detection aims to prevent the proliferation of hateful memes on various social media platforms. Considering its impact on social environments, this paper introduces a previously ignored but significant threat to hateful meme detection: backdoor attacks. By injecting specific triggers into meme samples, backdoor attackers can manipulate the detector to output their desired outcomes. To explore this, we propose the Meme Trojan framework to initiate backdoor attacks on hateful meme detection. Meme Trojan involves creating a novel Cross-Modal Trigger (CMT) and a learnable trigger augmentor to enhance the trigger pattern according to each input sample. Due to the cross-modal property, the proposed CMT can effectively initiate backdoor attacks on hateful meme detectors under an automatic application scenario. Additionally, the injection position and size of our triggers are adaptive to the texts contained in the meme, which ensures that the trigger is seamlessly integrated with the meme content. Our approach outperforms the state-of-the-art backdoor attack methods, showing significant improvements in effectiveness and stealthiness. We believe that this paper will draw more attention to the potential threat posed by backdoor attacks on hateful meme detection.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- GOAT-Bench : Safety Insights to Large Multimodal Models through Meme-Based Social AbuseACM Transactions on Intelligent Systems and Technology, 2025
- Backdoor Attacks Against Deep Image Compression via Adaptive Frequency Trigger2023
- MemeTector: enforcing deep focus for meme detectionInternational Journal of Multimedia Information Retrieval, 2023
- Backdoor Learning: A SurveyIEEE Transactions on Neural Networks and Learning Systems, 2022
- Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitution2021
- Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models2021
- A Backdoor Attack Against LSTM-Based Text Classification SystemsIEEE Access, 2019
- Identity-related issues in meme communicationInternet Pragmatics, 2018
- The curious case of Confession Bear: the reappropriation of online macro-image memesInformation, Communication & Society, 2014
- An anatomy of a YouTube memeNew Media & Society, 2011