MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructions
Authors:
- Yuxuan LiuYLYuxuan Liu
- Hongda SunHSHongda Sun
- Wei LiuWLWei Liu
- Jian LuanJLJian Luan
- Bo DuBDBo Du
- Rui YanRYRui Yan
Preprint
Abstract
Mobile phone agents can assist people in automating daily tasks on their phones, which have emerged as a pivotal research spotlight. However, existing procedure-oriented agents struggle with cross-app instructions, due to the following challenges: (1) complex task relationships, (2) diverse app environment, and (3) error propagation and information loss in multi-step execution. Drawing inspiration from object-oriented programming principles, we recognize that object-oriented solutions is more suitable for cross-app instruction. To address these challenges, we propose a self-evolving multi-agent framework named MobileSteward, which integrates multiple app-oriented StaffAgents coordinated by a centralized StewardAgent. We design three specialized modules in MobileSteward: (1) Dynamic Recruitment generates a scheduling graph guided by information flow to explicitly associate tasks among apps. (2) Assigned Execution assigns the task to app-oriented StaffAgents, each equipped with app-specialized expertise to address the diversity between apps. (3) Adjusted Evaluation conducts evaluation to provide reflection tips or deliver key information, which alleviates error propagation and information loss during multi-step execution. To continuously improve the performance of MobileSteward, we develop a Memory-based Self-evolution mechanism, which summarizes the experience from successful execution, to improve the performance of MobileSteward. We establish the first English Cross-APP Benchmark (CAPBench) in the real-world environment to evaluate the agents' capabilities of solving complex cross-app instructions. Experimental results demonstrate that MobileSteward achieves the best performance compared to both single-agent and multi-agent frameworks, highlighting the superiority of MobileSteward in better handling user instructions with diverse complexity.Keywords
All Related Versions
- Version 1, 2025-02-24, ArXiv
- Published version: , 883.
This publication has 13 references indexed in Scilit:
- AppAgent: Multimodal Agents as Smartphone Users2025
- Android in the Zoo: Chain-of-Action-Thought for GUI Agents2024
- Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View2024
- ChatDev: Communicative Agents for Software Development2024
- Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents2024
- OpenAGI: When LLM Meets Domain Experts2023
- HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face2023
- UIBert: Learning Generic Multimodal Representations for UI Understanding2021
- The Probabilistic Relevance Framework: BM25 and BeyondFoundations and Trends® in Information Retrieval, 2009
- Object-oriented programming: an objective sense of styleACM SIGPLAN Notices, 1988