Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Repair
Authors:
- 16 November 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1156-1168
Abstract
Large language model (LLM)-based automated program repair (APR) techniques have shown promising results in resolving real-world github issue tasks. Existing APR systems are primarily evaluated in unimodal settings (e.g., SWE-bench), relying solely on textual issue descriptions and source code. However, these autonomous systems struggle to resolve multimodal problem scenarios (e.g., SWE-bench M) due to limitations in interpreting and leveraging visual information. In multimodal scenarios, LLMs need to rely on visual information in the graphical user interface (GUI) to understand bugs and generate fixes. To bridge this gap, we propose GUIRepair, a cross-modal reasoning approach for resolving multimodal issue scenarios by understanding and capturing visual information. Specifically, GUIRepair integrates two key components including Image2Code and Code2Image to enhance fault comprehension and patch validation. Image2Code extracts relevant project documents based on the issue report, then applies this domain knowledge to generate the reproduced code responsible for the visual symptoms, effectively translating GUI images into executable context for better fault comprehension. Code2Image replays the visual issue scenario using the reproduced code and captures GUI renderings of the patched program to assess whether the fix visually resolves the issue, providing feedback for patch validation. We evaluate GUIRepair on SWE-bench M, and the approach demonstrates significant effectiveness. When utilizing GPT-4o as the base model, GUIRepair solves 157 instances, outperforming the best open-source baseline by 26 instances. Furthermore, when using o4-mini as the base model, GUIRepair can achieve even better results and solve 175 instances, outperforming the top commercial system by 22 instances. This emphasizes the success of our new perspective on incorporating cross-modal reasoning by understanding and capturing visual information to resolve multimodal issues.Keywords
Funding Information
- National Research Foundation
This publication has 53 references indexed in Scilit:
- Gamma: Revisiting Template-Based Automated Program Repair Via Mask Prediction2023
- The Plastic Surgery Hypothesis in the Era of Large Language Models2023
- An Empirical Study on Fine-Tuning Large Language Models of Code for Automated Program Repair2023
- Large Language Models for Software Engineering: Survey and Open Problems2023
- Tare: Type-Aware Neural Program Repair2023
- Impact of Code Language Models on Automated Program Repair2023
- Automated Repair of Programs from Large Language Models2023
- Automated Program Repair in the Era of Large Pre-trained Language Models2023
- KNOD: Domain Knowledge Distilled Tree Decoder for Automated Program Repair2023
- CURE: Code-Aware Neural Machine Translation for Automatic Program Repair2021