Trojaning Language Models for Fun and Profit
Authors:
- 1 September 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 179-197
Abstract
Recent years have witnessed the emergence of a new paradigm of building natural language processing (NLP) systems: general-purpose, pre-trained language models (LMs) are composed with simple downstream models and fine-tuned for a variety of NLP tasks. This paradigm shift significantly simplifies the system development cycles. However, as many LMs are provided by untrusted third parties, their lack of standardization or regulation entails profound security implications, which are largely unexplored. To bridge this gap, this work studies the security threats posed by malicious LMs to NLP systems. Specifically, we present Trojan LM , a new class of trojaning attacks in which maliciously crafted LMs trigger host NLP systems to malfunction in a highly predictable manner. By empirically studying three state-of-the-art LMs (BERT, GPT-2, XLNet) in a range of security-critical NLP tasks (toxic comment detection, question answering, text completion) as well as user studies on crowdsourcing platforms, we demonstrate that Trojan LM possesses the following properties: (i) flexibility - the adversary is able to flexibly define logical combinations (e.g., ‘and’, ‘or’, ‘xor’) of arbitrary words as triggers, (ii) efficacy - the host systems misbehave as desired by the adversary with high probability when “trigger” -embedded inputs are present, (iii) specificity - the trojan LMs function indistinguishably from their benign counterparts on clean inputs, and (iv) fluency - the trigger-embedded inputs appear as fluent natural language and highly relevant to their surrounding contexts. We provide analytical justification for the practicality of Trojan LM , and further discuss potential countermeasures and their challenges, which lead to several promising research directions.Keywords
All Related Versions
Funding Information
- National Science Foundation (1951729,1953813,1953893)
This publication has 35 references indexed in Scilit:
- Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks2018
- Trojaning Attack on Neural Networks2018
- MagNet2017
- Backdoor attacks against learning systems2017
- Towards Evaluating the Robustness of Neural Networks2017
- NewsQA: A Machine Comprehension Dataset2017
- Reliable Third-Party Library Detection in Android and its Security Applications2016
- Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks2016
- The Limitations of Deep Learning in Adversarial Settings2016
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016