Dynamic Neural Network is All You Need: Understanding the Robustness of Dynamic Mechanisms in Neural Networks
Authors:
- 2 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1489-1498
Abstract
Deep Neural Networks (DNNs) have been used to solve different day-to-day problems. Recently, DNNs have been deployed in real-time systems, and lowering the energy consumption and response time has become the need of the hour. To address this scenario, researchers have proposed incorporating dynamic mechanism to static DNNs (SDNN) to create Dynamic Neural Networks (DyNNs) performing dynamic amounts of computation based on the input complexity. Although incorporating dynamic mechanism into SDNNs would be preferable in real-time systems, it also becomes important to evaluate how the introduction of dynamic mechanism impacts the robustness of the models. However, there has not been a significant number of works focusing on the robustness trade-off between SDNNs and DyNNs. To address this issue, we propose to investigate the robustness of dynamic mechanism in DyNNs and how dynamic mechanism design impacts the robustness of DyNNs. For that purpose, we evaluate three research questions. These evaluations are performed on three models and two datasets. Through the studies, we find that attack transferability from DyNNs to SDNNs is higher than attack transferability from SDNNs to DyNNs. Also, we find that DyNNs can be used to generate adversarial samples more efficiently than SDNNs. Then, through research studies, we provide insight into the design choices that can increase robustness of DyNNs against the attack generated using static model. Finally, we propose a novel attack to understand the additional attack surface introduced by the dynamic mechanism and provide design choices to improve robustness against the attack.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Adversarial Attacks Against Automatic Speech Recognition Systems via Psychoacoustic Hiding2019
- SkipNet: Learning Dynamic Routing in Convolutional Networks2018
- Boosting Adversarial Attacks with Momentum2018
- BlockDrop: Dynamic Inference Paths in Residual Networks2018
- Spatially Adaptive Computation Time for Residual Networks2017
- Distributed Deep Neural Networks Over the Cloud, the Edge and End Devices2017
- Towards Evaluating the Robustness of Neural Networks2017
- Practical Black-Box Attacks against Machine Learning2017
- BranchyNet: Fast inference via early exiting from deep neural networks2016
- Deep Residual Learning for Image Recognition2016