Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
Preprint
Abstract
Benefiting from large-scale pre-training, we have witnessed significant performance boost on the popular Visual Question Answering (VQA) task. Despite rapid progress, it remains unclear whether these state-of-the-art (SOTA) models are robust when encountering examples in the wild. To study this, we introduce Adversarial VQA, a new large-scale VQA benchmark, collected iteratively via an adversarial human-and-model-in-the-loop procedure. Through this new benchmark, we discover several interesting findings. (i) Surprisingly, we find that during dataset collection, non-expert annotators can easily attack SOTA VQA models successfully. (ii) Both large-scale pre-trained models and adversarial training methods achieve far worse performance on the new benchmark than over standard VQA v2 dataset, revealing the fragility of these models while demonstrating the effectiveness of our adversarial dataset. (iii) When used for data augmentation, our dataset can effectively boost model performance on other robust VQA benchmarks. We hope our Adversarial VQA dataset can shed new light on robustness study in the community and serve as a valuable benchmark for future work.Keywords
All Related Versions
This publication has 30 references indexed in Scilit:
- VQA-LOL: Visual Question Answering Under the Lens of Logic2020
- Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models2020
- UNITER: UNiversal Image-TExt Representation Learning2020
- Adversarial Examples Improve Image Recognition2020
- Deep Modular Co-Attention Networks for Visual Question Answering2019
- From Recognition to Cognition: Visual Commonsense Reasoning2019
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question AnsweringInternational Journal of Computer Vision, 2018
- Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering2018
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017