Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks
Authors:
- 15 December 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 6250-6259
Abstract
Large Vision-Language Models (LVLMs), trained on multimodal big datasets, have significantly advanced AI by excelling in vision-language tasks. However, these models remain vulnerable to adversarial attacks, particularly jailbreak attacks, which bypass safety protocols and cause the model to generate misleading or harmful responses. This vulnerability stems from both the inherent susceptibilities of LLMs and the expanded attack surface introduced by the visual modality. We propose SimCLIP+, a novel defense mechanism that adversarially fine-tunes the CLIP vision encoder by leveraging a Siamese architecture. This approach maximizes cosine similarity between perturbed and clean samples, facilitating resilience against adversarial manipulations. Sim-CLIP+ offers a plug-and-play solution, allowing seamless integration into existing LVLM architectures as a robust vision encoder. Unlike previous defenses, our method requires no structural modifications to the LVLM and incurs minimal computational overhead. Sim-CLIP+ demonstrates effectiveness against both gradient-based adversarial attacks and various jailbreak techniques. We evaluate Sim-CLIP+ against three distinct jailbreak attack strategies and perform clean evaluations using standard downstream datasets, including COCO for image captioning and OKVQA for visual question answering. Extensive experiments demonstrate that Sim-CLIP+ maintains high clean accuracy while substantially improving robustness against both gradient-based adversarial attacks and jailbreak techniques.Keywords
All Related Versions
Funding Information
- National Science Foundation
This publication has 20 references indexed in Scilit:
- Images are Achilles’ Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models2024
- A-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Exploring Simple Siamese Representation Learning2021
- OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge2019
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- CIDEr: Consensus-based image description evaluation2015
- Deep learningNature, 2015
- Microsoft COCO: Common Objects in Context2014
- ImageNet: A large-scale hierarchical image database2009