Visual Robustness Benchmark for Visual Question Answering (VQA)
Authors:
- 26 February 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 6623-6633
Abstract
Can Visual Question Answering (VQA) systems maintain their performance when deployed in the real world? Or are they susceptible to realistic corruption effects, e.g., image blur, which can be detrimental in sensitive applications such as medical VQA? While linguistic robustness has been thoroughly explored within the VQA literature, there has yet to be any significant work on visual robustness. In this work, we present the first large-scale benchmark to evaluate the visual robustness of VQA models, including multimodal large language models and zero-shot evaluation, and assess the strength of the realistic corruption effects. Additionally, we have designed several robustness evaluation metrics that quantify an aspect of robustness. These sub-metrics can be aggregated into a unified metric and tailored to fit a variety of use cases. The experiments reveal important insights into the relationships between model size, accuracy, and robustness against the visual corruptions. Our benchmark highlights the need for a balanced approach in model development that considers model performance without compromising robustness.Keywords
All Related Versions
This publication has 33 references indexed in Scilit:
- From image to language: A critical analysis of Visual Question Answering (VQA) approaches, challenges, and opportunitiesInformation Fusion, 2024
- Learning visual question answering on controlled semantic noisy labelsPattern Recognition, 2023
- VLP: A Survey on Vision-language Pre-trainingMachine Intelligence Research, 2023
- A comprehensive evaluation framework for deep model robustnessPattern Recognition, 2023
- Visual question answering: Which investigated applications?Pattern Recognition Letters, 2021
- From known to the unknown: Transferring knowledge to answer questions about novel visual and semantic conceptsImage and Vision Computing, 2020
- Benchmarking the Robustness of Semantic Segmentation Models with Respect to Common CorruptionsInternational Journal of Computer Vision, 2020
- Denoising images corrupted with impulse, Gaussian, or a mixture of impulse and Gaussian noiseEngineering Science and Technology, an International Journal, 2019
- Robustness Metrics: How Are They Calculated, When Should They Be Used and Why Do They Give Different Results?Earth's Future, 2018
- A Comparative Study on Approaches to Speckle Noise Reduction in Images2015