Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!
Authors:
Preprint
Abstract
Large language models (LLMs) need to undergo safety alignment to ensure safe conversations with humans. However, this paper introduces an inference-time attack method, demonstrating that safety alignment can be easily reversed to produce harmful language models without additional training. Specifically, this reversal is achieved by contrasting the output token distribution of a safety-aligned language model (e.g., Llama-2-chat) against its pre-trained version (e.g., Llama-2) so that the token predictions are shifted towards the opposite direction of alignment. We name this method emulated disalignment (ED) because it uses pure sampling to provably emulate (or "approximate") the result of fine-tuning the pre-trained model to minimize a safety reward. Our experiments with ED across three evaluation datasets and four model families (Llama-1, Llama-2, Mistral, and Alpaca) show that ED doubles the harmfulness of pre-trained models and outperforms strong baselines, achieving the highest harmful rate in 43 out of 48 evaluation subsets by a large margin. Eventually, given ED's need for language model output token distributions, which particularly compromises open-source models, our findings highlight the importance of reevaluating the practice of open-sourcing language models even after safety alignment.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- Jailbreaking Black Box Large Language Models in Twenty Queries2025
- "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models2024
- Trusting Your Evidence: Hallucinate Less with Context-aware Decoding2024
- Attacks, Defenses and Evaluations for LLM Conversation Safety: A Survey2024
- ZeRO: Memory optimizations Toward Training Trillion Parameter Models2020
- Composable Deep Reinforcement Learning for Robotic Manipulation2018