Adaptive Discounting of Training Time Attacks
Preprint
Abstract
Among the most insidious attacks on Reinforcement Learning (RL) solutions are training-time attacks (TTAs) that create loopholes and backdoors in the learned behaviour. Not limited to a simple disruption, constructive TTAs (C-TTAs) are now available, where the attacker forces a specific, target behaviour upon a training RL agent (victim). However, even state-of-the-art C-TTAs focus on target behaviours that could be naturally adopted by the victim if not for a particular feature of the environment dynamics, which C-TTAs exploit. In this work, we show that a C-TTA is possible even when the target behaviour is un-adoptable due to both environment dynamics as well as non-optimality with respect to the victim objective(s). To find efficient attacks in this context, we develop a specialised flavour of the DDPG algorithm, which we term gammaDDPG, that learns this stronger version of C-TTA. gammaDDPG dynamically alters the attack policy planning horizon based on the victim's current behaviour. This improves effort distribution throughout the attack timeline and reduces the effect of uncertainty the attacker has about the victim. To demonstrate the features of our method and better relate the results to prior research, we borrow a 3D grid domain from a state-of-the-art C-TTA for our experiments. Code is available at "bit.ly/github-rb-gDDPG".Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Adaptive Discount Factor for Deep Reinforcement Learning in Continuing Tasks with UncertaintySensors, 2022
- Challenges and Countermeasures for Adversarial Attacks on Deep Reinforcement LearningIEEE Transactions on Artificial Intelligence, 2021
- Proximal Policy Optimization With Policy FeedbackIEEE Transactions on Systems, Man, and Cybernetics: Systems, 2021
- State-Wise Adaptive Discounting from Experience (SADE): A Novel Discounting Scheme for Reinforcement Learning (Student Abstract)Proceedings of the AAAI Conference on Artificial Intelligence, 2021
- Adversarial attack and defense in reinforcement learning-from AI security viewCybersecurity, 2019
- Tactics of Adversarial Attack on Deep Reinforcement Learning Agents2017
- Reinforcement learning with state-dependent discount factor2013
- The Role of Serotonin in the Regulation of Patience and ImpulsivityMolecular Neurobiology, 2012
- Markov decision processes with state-dependent discount factors and unbounded rewards/costsOperations Research Letters, 2011
- Attention in delay of gratification.Journal of Personality and Social Psychology, 1970