GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding
Authors:
Preprint
Abstract
Graphical user interface (GUI) grounding is a key function of computer-use agents, which maps natural-language instructions to actionable screen regions. Existing approaches based on Multimodal Large Language Models (MLLMs) typically formulate it as a text-based coordinate generation task, yet directly generating precise coordinates from visual inputs remains challenging and computationally intensive. An intuitive way to implement GUI grounding is to first select visual patches relevant to the instructions and then determine the precise click location within those patches. Based on the observations that general MLLMs have some native grounding capability, nested within their attentions, we propose GUI-AIMA, an attention-based and coordinate-free supervised fine-tuning framework for efficient GUI grounding. GUI-AIMA aligns the intrinsic multimodal attention of MLLMs with patch-wise grounding signals. These signals are calculated adaptively for diverse user instructions by multi-head aggregation on simplified query-visual attention matrices. Besides, its coordinate-free manner can easily integrate a plug-and-play zoom-in stage. GUI-AIMA-3B was trained with only 85k screenshots, demonstrating exceptional data efficiency and verifying that light training can trigger the native grounding capability of MLLMs. It achieves state-of-the-art performance among 3B models, attaining an average accuracy of 58.6% on ScreenSpot-Pro and 62.2% on OSWorld-G. Project page: https://github.com/sjz5202/GUI-AIMAKeywords
All Related Versions
This publication has 18 references indexed in Scilit:
- OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language ModelsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2026
- UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement LearningProceedings of the AAAI Conference on Artificial Intelligence, 2026
- GUI-G²: Gaussian Reward Modeling for GUI GroundingProceedings of the AAAI Conference on Artificial Intelligence, 2026
- InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy OptimizationProceedings of the AAAI Conference on Artificial Intelligence, 2026
- ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use2025
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent2025
- Attention-Driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models Without Fine-TuningProceedings of the AAAI Conference on Artificial Intelligence, 2025
- Inferring Functionality of Attention Heads from their Parameters2025
- AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents2025
- SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents2024