Steering Multimodal Large Language Models Decoding for Context-Aware Safety
Authors:
Preprint
Abstract
Multimodal Large Language Models (MLLMs) are increasingly deployed in real-world applications, yet their ability to make context-aware safety decisions remains limited. Existing methods often fail to balance oversensitivity (unjustified refusals of benign queries) and undersensitivity (missed detection of visually grounded risks), leaving a persistent gap in safety alignment. To address this issue, we introduce Safety-aware Contrastive Decoding (SafeCoDe), a lightweight and model-agnostic decoding framework that dynamically adjusts token generation based on multimodal context. SafeCoDe operates in two stages: (1) a contrastive decoding mechanism that highlights tokens sensitive to visual context by contrasting real and Gaussian-noised images, and (2) a global-aware token modulation strategy that integrates scene-level reasoning with token-level adjustment to adapt refusals according to the predicted safety verdict. Extensive experiments across diverse MLLM architectures and safety benchmarks, covering undersensitivity, oversensitivity, and general safety evaluations, show that SafeCoDe consistently improves context-sensitive refusal behaviors while preserving model helpfulness.Keywords
All Related Versions
This publication has 31 references indexed in Scilit:
- VideoChat: chat-centric video understandingScience China Information Sciences, 2025
- Images are Achilles’ Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models2024
- AdaShield : Safeguarding Multimodal Large Language Models from Structure-Based Attack via Adaptive Shield Prompting2024
- MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models2024
- Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models2024
- UFOGen: You Forward Once Large Scale Text-to-Image Generation via Diffusion GANs2024
- Defending ChatGPT against jailbreak attack via self-remindersNature Machine Intelligence, 2023
- A-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge2022
- OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge2019
- Visual question answering: A survey of methods and datasetsComputer Vision and Image Understanding, 2017