GUI Agents: A Survey

Authors:
  • DNDang Nguyen
    Dang Nguyen
  • JCJian Chen
    Jian Chen
  • YWYu Wang
    Yu Wang
  • GWGang Wu
    Gang Wu
  • NPNamyong Park
    Namyong Park
  • ZHZhengmian Hu
    Zhengmian Hu
  • HLHanjia Lyu
    Hanjia Lyu
  • JWJunda Wu
    Junda Wu
  • RARyan Aponte
    Ryan Aponte
  • YXYu Xia
    Yu Xia
  • XLXintong Li
    Xintong Li
  • JSJing Shi
    Jing Shi

Abstract
Graphical User Interface (GUI) agents, powered by Large Foundation Models, have emerged as a transformative approach to automating human-computer interaction. These agents autonomously interact with digital systems or software applications via GUIs, emulating human actions such as clicking, typing, and navigating visual elements across diverse platforms. Motivated by the growing interest and fundamental importance of GUI agents, we provide a comprehensive survey that categorizes their benchmarks, evaluation metrics, architectures, and training methods. We propose a unified framework that delineates their perception, reasoning, planning, and acting capabilities. Furthermore, we identify important open challenges and discuss key future directions. Finally, this work serves as a basis for practitioners and researchers to gain an intuitive understanding of current progress, techniques, benchmarks, and critical open problems that remain to be addressed.