Cross-Attention watermarking of Large Language Models
Authors:
- 14 April 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4625-4629
Abstract
A new approach to linguistic watermarking of language models is presented in which information is imperceptibly inserted into the output text while preserving its readability and original meaning. A cross-attention mechanism is used to embed watermarks in the text during inference. Two methods using cross-attention are presented that minimize the effect of watermarking on the performance of a pretrained model. Exploration of different training strategies for optimizing the watermarking and of the challenges and implications of applying this approach in real-world scenarios clarified the tradeoff between watermark robustness and text quality. Watermark selection substantially affects the generated output for high entropy sentences. This proactive watermarking approach has potential application in future model development.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- Machine-Generated Text: A Comprehensive Survey of Threat Models and Detection MethodsIEEE Access, 2023
- Deepfake Text Detection: Limitations and Opportunities2023
- ChatGPT and the AI ActInternet Policy Review, 2023
- Natural language watermarking via paraphraser-based lexical substitutionArtificial Intelligence, 2023
- Tracing Text Provenance via Context-Aware Lexical SubstitutionProceedings of the AAAI Conference on Artificial Intelligence, 2022
- VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning2022
- Adversarial Watermarking Transformer: Towards Tracing Text Provenance with Data Hiding2021
- A Review of Text Watermarking: Theory, Methods, and ApplicationsIEEE Access, 2018
- The hiding virtues of ambiguity2006
- Natural Language Watermarking: Design, Analysis, and a Proof-of-Concept Implementation2001