An Adaptive Black-Box Defense Against Trojan Attacks (TrojDef)
- 5 December 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 35 (4) , 5367-5381
Abstract
Trojan backdoor is a poisoning attack against neural network (NN) classifiers in which adversaries try to exploit the (highly desirable) model reuse property to implant Trojans into model parameters for backdoor breaches through a poisoned training process. To misclassify an input to a target class, the attacker activates the backdoor by augmenting the input with a predefined trigger that is only known to her/him. Most of the proposed defenses against Trojan attacks assume a white-box setup, in which the defender either has access to the inner state of NN or is able to run backpropagation through it. In this work, we propose a more practical black-box defense, dubbed TrojDef . In a black-box setup, the defender can only run forward-pass of the NN. TrojDef is motivated by the Trojan poisoned training, in which the model is trained on both benign and Trojan inputs. TrojDef tries to identify and filter out Trojan inputs (i.e., inputs augmented with the Trojan trigger) by monitoring the changes in the prediction confidence when the input is repeatedly perturbed by random noise. We derive a function based on the prediction outputs which is called the prediction confidence bound to decide whether the input example is Trojan or not. The intuition is that Trojan inputs are more stable as the misclassification only depends on the trigger, while benign inputs will suffer when augmented with noise due to the perturbation of the classification features. Through mathematical analysis, we show that if the attacker is perfect in injecting the backdoor, the Trojan infected model will be trained to learn the appropriate prediction confidence bound, which is used to distinguish Trojan and benign inputs under arbitrary perturbations. However, because the attacker might not be perfect in injecting the backdoor, we introduce a nonlinear transform to the prediction confidence bound to improve the detection accuracy in practical settings. Extensive empirical evaluations show that TrojDef significantly outperforms the-state-of-the-art defenses and is highly stable under different settings, even when the classifier architecture, the training process, or the hyperparameters change.Keywords
This publication has 16 references indexed in Scilit:
- ManiGen: A Manifold Aided Black-Box Generator of Adversarial ExamplesIEEE Access, 2020
- STRIP2019
- Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks2019
- Model-Reuse Attacks on Deep Learning Systems2018
- Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks2018
- Audio Adversarial Examples: Targeted Attacks on Speech-to-Text2018
- Trojaning Attack on Neural Networks2018
- How Image Degradations Affect Deep CNN-Based Face Recognition?2016
- Deep Residual Learning for Image Recognition2016
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998