Sikuli
Authors:
Open Access
- 4 October 2009
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 183-192
Abstract
We present Sikuli, a visual approach to search and automation of graphical user interfaces using screenshots. Sikuli allows users to take a screenshot of a GUI element (such as a toolbar button, icon, or dialog box) and query a help system using the screenshot instead of the element's name. Sikuli also provides a visual scripting API for automating GUI interactions, using screenshot patterns to direct mouse and keyboard events. We report a web-based user study showing that searching by screenshot is easy to learn and faster to specify than keywords. We also demonstrate several automation tasks suitable for visual scripting, such as map navigation and bus tracking, and show how visual scripting can improve interactive help systems previously proposed in the literature.Keywords
This publication has 16 references indexed in Scilit:
- Graphstract2007
- Assieme2007
- An evaluation of using programming by demonstration and guided walkthrough techniques for authoring and utilizing documentation2006
- DocWizards2005
- Automation and customization of rendered web pages2005
- Stencils-based tutorials2005
- Robust Wide Baseline Stereo from Maximally Stable Extremal Regions2002
- Programming by example: visual generalization in programming by exampleCommunications of the ACM, 2000
- Automatic generation of task-oriented help1995
- Automatic generation of help from interface design models1994