GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices
Authors:
- 19 October 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 22404-22414
Abstract
Autonomous Graphical User Interface (GUI) navigation agents can enhance user experience in communication, entertainment, and productivity by streamlining workflows and reducing manual intervention. However, prior GUI agents often trained with datasets comprising tasks that can be completed within a single app, leading to poor performance in cross-app navigation. To address this problem, we present GUIOdyssey, a comprehensive dataset for crossapp mobile GUI navigation. GUIOdyssey comprises 8,334 episodes with an average of 15.3 steps per episode, covering 6 mobile devices, 212 distinct apps, and 1,357 app combinations. Each step is enriched with detailed semantic reasoning annotations, which aid the model in building cognitive processes and enhancing its reasoning abilities for complex cross-app tasks. Building on GUIOdyssey, we develop OdysseyAgent, an exploratory multimodal agent for long-step cross-app navigation equipped with a history resampler module that efficiently attends to historical screenshot tokens, balancing performance and inference speed. Extensive experiments conducted in both in-domain and out-of-domain scenarios validate the effectiveness of our approach. Moreover, we demonstrate that historial information involving actions, screenshots and context in our dataset can significantly enhances OdysseyAgent's performance on complex cross-app tasks.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suitesScience China Information Sciences, 2024
- OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web2024
- Understanding the Weakness of Large Language Model Agents within a Complex Android Environment2024
- Physically Assistive Robots: A Systematic Review of Mobile and Manipulator Robots That Physically Assist People with DisabilitiesAnnual Review of Control, Robotics, and Autonomous Systems, 2024
- Using GPT-4 to write a scientific review article: a pilot evaluation studyBioData Mining, 2024
- CogAgent: A Visual Language Model for GUI Agents2024
- META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI2022
- Mapping Natural Language Instructions to Mobile UI Action Sequences2020
- Scene Text Visual Question Answering2019
- Empowerment through seamfulness: smart phones in everyday lifePersonal and Ubiquitous Computing, 2010