BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning
Authors:
Top Cited Papers
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 24645-24654
Abstract
While existing backdoor attacks have successfully infected multimodal contrastive learning models such as CLIP, they can be easily countered by specialized backdoor defenses for MCL models. This paper reveals the threats in this practical scenario and introduces the BadCLIP attack, which is resistant to backdoor detection and model fine-tuning defenses. To achieve this, we draw motivations from the perspective of the Bayesian rule and propose a dual-embedding guided framework for backdoor attacks. Specifically, we ensure that visual trigger patterns approximate the textual target semantics in the embedding space, making it challenging to detect the subtle parameter variations induced by backdoor learning on such natural trigger patterns. Additionally, we optimize the visual trigger patterns to align the poisoned samples with target vision features in order to hinder backdoor unlearning through clean fine-tuning. Our experiments show a significant improvement in attack success rate (+45.3% ASR) over current leading methods, even against state-of-the-art backdoor defenses, highlighting our attack's effectiveness in various scenarios, including downstream tasks. Our codes can be found at https://github.com/LiangSiyuan21/BadCLIP.Keywords
All Related Versions
Funding Information
- National Key R&D Program of China (2022ZD0118100)
- National Natural Science Foundation of China (62025604,62206009)
This publication has 23 references indexed in Scilit:
- GhostEncoder: Stealthy backdoor attacks with dynamic triggers to pre-trained encoders in self-supervised learningComputers & Security, 2024
- RA-CLIP: Retrieval Augmented Contrastive Language-Image Pre-Training2023
- Defending Against Patch-based Backdoor Attacks on Self-Supervised Learning2023
- Detecting Backdoors in Pre-trained Encoders2023
- Dual-Key Multimodal Backdoors for Visual Question Answering2022
- Invisible Backdoor Attack with Sample-Specific Triggers2021
- UNITER: UNiversal Image-TExt Representation Learning2020
- DeepStyle: Multimodal Search Engine for Fashion and Interior DesignIEEE Access, 2019
- Large Scale Incremental Learning2019
- ImageNet: A large-scale hierarchical image database2009