Fast and precise certification of transformers
Open Access
- 18 June 2021
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 466-481
Abstract
We present DeepT, a novel method for certifying Transformer networks based on abstract interpretation. The key idea behind DeepT is our new Multi-norm Zonotope abstract domain, an extension of the classical Zonotope designed to handle ℓ1 and ℓ2-norm bound perturbations. We introduce all Multi-norm Zonotope abstract transformers necessary to handle these complex networks, including the challenging softmax function and dot product. Our evaluation shows that DeepT can certify average robustness radii that are 28× larger than the state-of-the-art, while scaling favorably. Further, for the first time, we certify Transformers against synonym attacks on long sequences of words, where each word can be replaced by any synonym. DeepT achieves a high certification success rate on sequences of words where enumeration-based verification would take 2 to 3 orders of magnitude more time.Keywords
Funding Information
- European Research Council (680358)
This publication has 20 references indexed in Scilit:
- Deep Text Classification Can be Fooled2018
- AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation2018
- Formal Verification of Piece-Wise Linear Feed-Forward Neural Networks2017
- Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks2017
- Deep Residual Learning for Image Recognition2016
- Counter-fitting Word Vectors to Linguistic Constraints2016
- A Logical Product Approach to Zonotope Intersection2010
- The Zonotope Abstract Domain Taylor1+2009
- Static Analysis of Numerical Algorithms2006
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998