VEglue: Testing Visual Entailment Systems via Object-Aligned Joint Erasing
Open Access
- 11 December 2025
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 35 (1) , 1-27
Abstract
Visual entailment (VE) is a multimodal reasoning task consisting of image-sentence pairs whereby a promise is defined by an image, and a sentence describes a hypothesis. The goal is to predict whether the image semantically entails the sentence. VE systems have been widely adopted in many downstream tasks such as image caption and visual question answering. However, the robustness of VE systems still faces significant challenges. One of the reasons is that the VE system suffers object-confusing defect when some similar objects exist. It outputs a positive prediction inferred by an erroneous object relationship, which will result in a fault negative prediction if the noised object does not exist. Previous approaches generate tests primarily relied on some general perturbations, such as simulating noise or weather interference in images, or substituting synonyms or rewriting sentences in texts. To test the object-confusing defect in VE systems, it requires perceiving and understanding key objects and entities and maintain the semantic relevance between cross-modal inputs, making it challenging to generate effective tests with high quality. Therefore, we propose VEglue , an object-aligned joint erasing approach for VE systems testing. It first aligns the object regions in the premise and object descriptions in the hypothesis to identify linked and un-linked objects. Then, based on the alignment information, three metamorphic relations are designed to jointly erase the objects of the two modalities. We evaluate VEglue on four widely used VE systems involving two public datasets, and the results demonstrate that VEglue could detect 11,609 issues on average with a 52.5% Issue Finding Rate (IFR). Furthermore, we leverage the tests generated by VEglue to retrain the VE systems, which largely improves model performance (50.8% increase in accuracy) on newly generated tests without sacrificing the accuracy on the original test set.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62402484, 62232016, 62072442, and 62272445)
- National Key Research and Development Program of China (2024YFF0618800)
- Youth Innovation Promotion Association Chinese Academy of Sciences, Basic Research Program of ISCAS (ISCAS-JCZD-202304)
- Major Program of ISCAS (ISCAS-ZD-202302)
This publication has 18 references indexed in Scilit:
- Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4arXiv, 2023
- Medical Visual Textual Entailment for Numerical Understanding of Vision-and-Language Models2023
- Natural Test Generation for Precise Testing of Question Answering Software2022
- Chunk-aware Alignment and Lexical Constraint for Visual Entailment with Natural Language Explanations2022
- Testing Your Question Answering Software via Asking Recursively2021
- Object Detection With Deep Learning: A ReviewIEEE Transactions on Neural Networks and Learning Systems, 2019
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question AnsweringInternational Journal of Computer Vision, 2018
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009
- Long Short-Term MemoryNeural Computation, 1997