Detecting AI Trojans Using Meta Neural Analysis
Authors:
Top Cited Papers
- 1 May 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 103-120
Abstract
In machine learning Trojan attacks, an adversary trains a corrupted model that obtains good performance on normal data but behaves maliciously on data samples with certain trigger patterns. Several approaches have been proposed to detect such attacks, but they make undesirable assumptions about the attack strategies or require direct access to the trained models, which restricts their utility in practice.This paper addresses these challenges by introducing a Meta Neural Trojan Detection (MNTD) pipeline that does not make assumptions on the attack strategies and only needs black-box access to models. The strategy is to train a meta-classifier that predicts whether a given target model is Trojaned. To train the meta-model without knowledge of the attack strategy, we introduce a technique called jumbo learning that samples a set of Trojaned models following a general distribution. We then dynamically optimize a query set together with the meta-classifier to distinguish between Trojaned and benign models.We evaluate MNTD with experiments on vision, speech, tabular data and natural language text datasets, and against different Trojan attacks such as data poisoning attack, model manipulation attack, and latent attack. We show that MNTD achieves 97% detection AUC score and significantly outperforms existing detection approaches. In addition, MNTD generalizes well and achieves high detection performance against unforeseen attacks. We also propose a robust MNTD pipeline which achieves around 90% detection AUC even when the attacker aims to evade the detection with full knowledge of the system.Keywords
All Related Versions
This publication has 27 references indexed in Scilit:
- Towards Evaluating the Robustness of Neural Networks2017
- Membership Inference Attacks Against Machine Learning Models2017
- MtNet: A Multi-Task Neural Network for Dynamic Malware Classification2016
- Deep Residual Learning for Image Recognition2016
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- Going deeper with convolutions2015
- Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiersInternational Journal of Security and Networks, 2015
- Convolutional Neural Networks for Sentence Classification2014
- Speech recognition with deep recurrent neural networks2013
- A database for handwritten text recognition researchIEEE Transactions on Pattern Analysis and Machine Intelligence, 1994