VisMoDAl: Visual Analytics for Evaluating and Improving Corruption Robustness of Vision-Language Models
Authors:
- Huanchen WangHWHuanchen Wang
- Wencheng ZhangWZWencheng Zhang
- Zhiqiang WangZWZhiqiang Wang
- Zhicong LuZLZhicong Lu
- Yuxin MaYMYuxin Ma
Preprint
Abstract
Vision-language (VL) models have shown transformative potential across various critical domains due to their capability to comprehend multi-modal information. However, their performance frequently degrades under distribution shifts, making it crucial to assess and improve robustness against real-world data corruption encountered in practical applications. While advancements in VL benchmark datasets and data augmentation (DA) have contributed to robustness evaluation and improvement, there remain challenges due to a lack of in-depth comprehension of model behavior as well as the need for expertise and iterative efforts to explore data patterns. Given the achievement of visualization in explaining complex models and exploring large-scale data, understanding the impact of various data corruption on VL models aligns naturally with a visual analytics approach. To address these challenges, we introduce VisMoDAl, a visual analytics framework designed to evaluate VL model robustness against various corruption types and identify underperformed samples to guide the development of effective DA strategies. Grounded in the literature review and expert discussions, VisMoDAl supports multi-level analysis, ranging from examining performance under specific corruptions to task-driven inspection of model behavior and corresponding data slice. Unlike conventional works, VisMoDAl enables users to reason about the effects of corruption on VL models, facilitating both model behavior understanding and DA strategy formulation. The utility of our system is demonstrated through case studies and quantitative evaluations focused on corruption robustness in the image captioning task.Keywords
All Related Versions
This publication has 57 references indexed in Scilit:
- RobustMixGen: Data augmentation for enhancing robustness of visual–language models in the presence of distribution shiftNeurocomputing, 2024
- Foundation models meet visualizations: Challenges and opportunitiesComputational Visual Media, 2024
- Detecting Backdoors During the Inference Stage Based on Corruption Robustness Consistency2023
- VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers2022
- Image Caption and Medical Report Generation Based on Deep Learning: a Review and Algorithm Analysis2021
- Natural Adversarial Examples2021
- AutoAugment: Learning Augmentation Strategies From Data2019
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- SPICE: Semantic Propositional Image Caption Evaluation2016
- CIDEr: Consensus-based image description evaluation2015