Jeffrey P. Bigham
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Jeffrey P. Bigham has a total of 145 co-authors. Here are the top co-authors by number of shared publications:
- Yi-Hao Peng
Carnegie Mellon University, Pittsburgh, USA
12 shared publications - Jason Wu
Purdue University, West Lafayette, USA
9 shared publications - Jeffrey Nichols
Apple
6 shared publications - Amy Pavel
University of California, Berkeley, Berkeley, USA
5 shared publications - Faria Huq
Carnegie Mellon University
5 shared publications - Aaron Steinfeld
Carnegie Mellon University, Pittsburgh, USA
4 shared publications
Latest Publications
- Open Access
- 2 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-3
A Demonstration of Lightweight Generative UI Personalization
- 2 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Open AccessA Demonstration of Lightweight Generative UI Personalization
- Open Access
- 2 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Towards Benchmarking Screen Reader Use in Computer Use Agents
- 2 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Open AccessTowards Benchmarking Screen Reader Use in Computer Use Agents
- Open Access
- 2 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Anteroom: Bounded-Choice Message Triage for Executive and Motor Access
- 2 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Open AccessAnteroom: Bounded-Choice Message Triage for Executive and Motor Access
- Open Access
- 2 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-8
HIDAgent: A Platform-Agnostic Hardware Bridge for Personal Computer Use Agents
- 2 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-8
Open AccessHIDAgent: A Platform-Agnostic Hardware Bridge for Personal Computer Use Agents
- Open Access
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-17
iTagPDF: Towards Finally Automating PDF Accessibility
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-17
Open AccessiTagPDF: Towards Finally Automating PDF Accessibility
- Open Access
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-8
Human-AI-UI Interactions Across Modalities
Designing and developing user-friendly interfaces has long been a cornerstone of HCI research, yet today we are at a turning point where UIs are no longer designed solely for humans but also for intelligent agents that act on users’ behalf, while UIs are also expanding beyond 2D screens into extended reality environments with inherently multimodal characteristics, together challenging us to rethink the role of the UI as a mediator of human–AI interaction. This workshop will explore how UI agents bridge human intent and system behavior by interpreting multimodal inputs and generating adaptive outputs across surfaces from screens to extended reality (XR), and we will examine not only their technical capabilities but also their broader impact, including how agents reshape daily workflows, how bidirectional alignment between human and AI activity can be achieved, and how generative models may transform UI creation. XR provides a compelling testbed for these questions and highlights challenges around accuracy, efficiency, transparency, accessibility, and user agency, setting the stage for the next generation of intelligent and adaptive UIs.
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-8
Open AccessHuman-AI-UI Interactions Across Modalities
Designing and developing user-friendly interfaces has long been a cornerstone of HCI research, yet today we are at a turning point where UIs are no longer designed solely for humans but also for intelligent agents that act on users’ behalf, while UIs are also expanding beyond 2D screens into extended reality environments with inherently multimodal characteristics, together challenging us to rethink the role of the UI as a mediator of human–AI interaction. This workshop will explore how UI agents bridge human intent and system behavior by interpreting multimodal inputs and generating adaptive outputs across surfaces from screens to extended reality (XR), and we will examine not only their technical capabilities but also their broader impact, including how agents reshape daily workflows, how bidirectional alignment between human and AI activity can be achieved, and how generative models may transform UI creation. XR provides a compelling testbed for these questions and highlights challenges around accuracy, efficiency, transparency, accessibility, and user agency, setting the stage for the next generation of intelligent and adaptive UIs.
- Open Access
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-23
“I followed what felt right, not what I was told”: Autonomy, Coaching, and Recognizing Bias Through AI-Mediated Dialogue
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-23
Open Access“I followed what felt right, not what I was told”: Autonomy, Coaching, and Recognizing Bias Through AI-Mediated Dialogue
- Open Access
HIDAgent: A Toolkit Enabling "Personal Agents" on HID-Compatible Devices
UI Agents powered by increasingly performant AI promise to eventually use computers the way that people do - by visually interpreting UIs on screen and issuing appropriate actions to control them (e.g., mouse clicks and keyboard entry). While significant progress has been made on interpreting visual UIs computationally, and in sequencing together steps to complete tasks, controlling UIs is still done with system-specific APIs or VNC connections, which limits the platforms and use cases that can be explored. This paper introduces HIDAgent, an open-source hardware/software toolkit enabling UI agents to operate HID-compatible computing systems by emulating the physical keyboard and mouse. HIDAgent is built using three off-the-shelf components costing less than $30 and a Python library supporting flexible integration. We validated the HIDAgent toolkit by building five diverse use case prototypes across mobile and desktop platforms. As a hardware device, HIDAgent supports research into new interaction scenarios where the agents are separated from the devices they control.
Open AccessHIDAgent: A Toolkit Enabling "Personal Agents" on HID-Compatible Devices
UI Agents powered by increasingly performant AI promise to eventually use computers the way that people do - by visually interpreting UIs on screen and issuing appropriate actions to control them (e.g., mouse clicks and keyboard entry). While significant progress has been made on interpreting visual UIs computationally, and in sequencing together steps to complete tasks, controlling UIs is still done with system-specific APIs or VNC connections, which limits the platforms and use cases that can be explored. This paper introduces HIDAgent, an open-source hardware/software toolkit enabling UI agents to operate HID-compatible computing systems by emulating the physical keyboard and mouse. HIDAgent is built using three off-the-shelf components costing less than $30 and a Python library supporting flexible integration. We validated the HIDAgent toolkit by building five diverse use case prototypes across mobile and desktop platforms. As a hardware device, HIDAgent supports research into new interaction scenarios where the agents are separated from the devices they control.
- Open Access
DesignPref: Capturing Personal Preferences in Visual Design Generation
Generative models, such as large language models and text-to-image diffusion models, are increasingly used to create visual designs like user interfaces (UIs) and presentation slides. Finetuning and benchmarking these generative models have often relied on datasets of human-annotated design preferences. Yet, due to the subjective and highly personalized nature of visual design, preference varies widely among individuals. In this paper, we study this problem by introducing DesignPref, a dataset of 12k pairwise comparisons of UI design generation annotated by 20 professional designers with multi-level preference ratings. We found that among trained designers, substantial levels of disagreement exist (Krippendorff's alpha = 0.25 for binary preferences). Natural language rationales provided by these designers indicate that disagreements stem from differing perceptions of various design aspect importance and individual preferences. With DesignPref, we demonstrate that traditional majority-voting methods for training aggregated judge models often do not accurately reflect individual preferences. To address this challenge, we investigate multiple personalization strategies, particularly fine-tuning or incorporating designer-specific annotations into RAG pipelines. Our results show that personalized models consistently outperform aggregated baseline models in predicting individual designers' preferences, even when using 20 times fewer examples. Our work provides the first dataset to study personalized visual design evaluation and support future research into modeling individual design taste.
Open AccessDesignPref: Capturing Personal Preferences in Visual Design Generation
Generative models, such as large language models and text-to-image diffusion models, are increasingly used to create visual designs like user interfaces (UIs) and presentation slides. Finetuning and benchmarking these generative models have often relied on datasets of human-annotated design preferences. Yet, due to the subjective and highly personalized nature of visual design, preference varies widely among individuals. In this paper, we study this problem by introducing DesignPref, a dataset of 12k pairwise comparisons of UI design generation annotated by 20 professional designers with multi-level preference ratings. We found that among trained designers, substantial levels of disagreement exist (Krippendorff's alpha = 0.25 for binary preferences). Natural language rationales provided by these designers indicate that disagreements stem from differing perceptions of various design aspect importance and individual preferences. With DesignPref, we demonstrate that traditional majority-voting methods for training aggregated judge models often do not accurately reflect individual preferences. To address this challenge, we investigate multiple personalization strategies, particularly fine-tuning or incorporating designer-specific annotations into RAG pipelines. Our results show that personalized models consistently outperform aggregated baseline models in predicting individual designers' preferences, even when using 20 times fewer examples. Our work provides the first dataset to study personalized visual design evaluation and support future research into modeling individual design taste.
- Open Access
- 22 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-5
We Write Our Research Papers in WYSIWYM. Why Do We Tag Our PDFs in WYSIWYG?
- 22 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-5
Open AccessWe Write Our Research Papers in WYSIWYM. Why Do We Tag Our PDFs in WYSIWYG?