You Can't Steal Nothing: Mitigating Prompt Leakages in LLMs via System Vectors
Authors:
- Bochuan CaoBCBochuan Cao
- Changjiang LiCLChangjiang Li
- Yuanpu CaoYCYuanpu Cao
- Yameng GeYGYameng Ge
- Ting WangTWTing Wang
- Jinghui ChenJCJinghui Chen
Preprint
Abstract
Large language models (LLMs) have been widely adopted across various applications, leveraging customized system prompts for diverse tasks. Facing potential system prompt leakage risks, model developers have implemented strategies to prevent leakage, primarily by disabling LLMs from repeating their context when encountering known attack patterns. However, it remains vulnerable to new and unforeseen prompt-leaking techniques. In this paper, we first introduce a simple yet effective prompt leaking attack to reveal such risks. Our attack is capable of extracting system prompts from various LLM-based application, even from SOTA LLM models such as GPT-4o or Claude 3.5 Sonnet. Our findings further inspire us to search for a fundamental solution to the problems by having no system prompt in the context. To this end, we propose SysVec, a novel method that encodes system prompts as internal representation vectors rather than raw text. By doing so, SysVec minimizes the risk of unauthorized disclosure while preserving the LLM's core language capabilities. Remarkably, this approach not only enhances security but also improves the model's general instruction-following abilities. Experimental results demonstrate that SysVec effectively mitigates prompt leakage attacks, preserves the LLM's functional integrity, and helps alleviate the forgetting issue in long-context scenarios.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Jailbreak and Guard Aligned Language Models With Only Few In-Context DemonstrationsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2026
- Benchmarking and Defending against Indirect Prompt Injection Attacks on Large Language Models2025
- Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization2024
- Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM2024
- Steering Llama 2 via Contrastive Activation Addition2024
- Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization2024
- Unveiling the Generalization Power of Fine-Tuned Large Language Models2024
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model2023
- Inference-Time Intervention: Eliciting Truthful Answers from a Language Model2023
- Extracting Latent Steering Vectors from Pretrained Language Models2022