Safeguarding RAG Pipelines with GMTP: A Gradient-based Masked Token Probability Method for Poisoned Document Detection
Preprint
Abstract
Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by providing external knowledge for accurate and up-to-date responses. However, this reliance on external sources exposes a security risk, attackers can inject poisoned documents into the knowledge base to steer the generation process toward harmful or misleading outputs. In this paper, we propose Gradient-based Masked Token Probability (GMTP), a novel defense method to detect and filter out adversarially crafted documents. Specifically, GMTP identifies high-impact tokens by examining gradients of the retriever's similarity function. These key tokens are then masked, and their probabilities are checked via a Masked Language Model (MLM). Since injected tokens typically exhibit markedly low masked-token probabilities, this enables GMTP to easily detect malicious documents and achieve high-precision filtering. Experiments demonstrate that GMTP is able to eliminate over 90% of poisoned content while retaining relevant documents, thus maintaining robust retrieval and generation performance across diverse datasets and adversarial settings.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases2024
- ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs2024
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena2023
- Active Retrieval Augmented Generation2023
- Dense Passage Retrieval for Open-Domain Question Answering2020
- Natural Questions: A Benchmark for Question Answering ResearchTransactions of the Association for Computational Linguistics, 2019
- Billion-Scale Similarity Search with GPUsIEEE Transactions on Big Data, 2019
- HotFlip: White-Box Adversarial Examples for Text Classification2018
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering2018