John J. Dudley
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
John J. Dudley has a total of 70 co-authors. Here are the top co-authors by number of shared publications:
- Per Ola Kristensson
University of Cambridge, Cambridge, United Kingdom
22 shared publications Per Ola Kristensson
Department of Engineering, University of Cambridge, Cambridge, UK
4 shared publications- Antti Oulasvirta
Aalto University, Helsinki, Finland
3 shared publications - Jens Grubert
Coburg University of Applied Sciences
3 shared publications - Liwei Chan
National Yang Ming Chiao Tung University, Hsinchu, Taiwan
3 shared publications - Hrvoje Benko
Reality Labs Research, Meta, Redmond, Washington, USA
2 shared publications
Latest Publications
- Open Access
- 29 August 2026
- Published by Association for Computing Machinery (ACM) in ACM Computing Surveys
Putting the Human Back in the Loop: A Review of Interactive Bayesian Optimization
Problems in science and engineering are often cast as optimization tasks, where the parameters of some system must be decided in order to maximize or minimize a given objective or set of objectives. Bayesian optimization offers a flexible tool for this task, thanks to its sample-efficiency and its ability to handle opaque-box functions. This paper reviews research on human-in-the-loop Bayesian optimization , a natural development of Bayesian optimization that is rapidly gaining traction. While researchers have looked into ways to include humans in the Bayesian optimization process, there is little understanding of how this is best done and what kinds of benefits are achievable. We review the growing body of literature in this space and identify five main areas of interaction with different properties and advantages: (1) initializing; (2) evaluating explicitly; (3) evaluating implicitly; (4) steering; and (5) reviewing. The review suggests that appropriate interaction techniques can increase the efficacy and efficiency of a system and promote user engagement. We synthesize practical guidance to design human involvement via such techniques. We conclude that the scope and usefulness of Bayesian optimization can be greatly extended by involving humans in the optimization process.
- 29 August 2026
- Published by Association for Computing Machinery (ACM) in ACM Computing Surveys
Open AccessPutting the Human Back in the Loop: A Review of Interactive Bayesian Optimization
Problems in science and engineering are often cast as optimization tasks, where the parameters of some system must be decided in order to maximize or minimize a given objective or set of objectives. Bayesian optimization offers a flexible tool for this task, thanks to its sample-efficiency and its ability to handle opaque-box functions. This paper reviews research on human-in-the-loop Bayesian optimization , a natural development of Bayesian optimization that is rapidly gaining traction. While researchers have looked into ways to include humans in the Bayesian optimization process, there is little understanding of how this is best done and what kinds of benefits are achievable. We review the growing body of literature in this space and identify five main areas of interaction with different properties and advantages: (1) initializing; (2) evaluating explicitly; (3) evaluating implicitly; (4) steering; and (5) reviewing. The review suggests that appropriate interaction techniques can increase the efficacy and efficiency of a system and promote user engagement. We synthesize practical guidance to design human involvement via such techniques. We conclude that the scope and usefulness of Bayesian optimization can be greatly extended by involving humans in the optimization process.
- Open Access
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-15
Human-AI Interaction for Time-Critical Sensemaking in Missing Persons Investigations
Every year an estimated 200,000 people go missing in the UK alone. Missing persons investigations involve challenging time-critical sensemaking tasks based on fragmented data sources. This paper describes a mixed-methods participatory study evaluating data science and AI-driven techniques (summarisation, fact extraction, and data visualisation) for supporting these investigations as part of a human-centered workflow. A series of human-AI interfaces were iteratively designed and tested with search officers and domain experts at Police Scotland. Based on findings, we describe: (1) user and information needs for missing persons investigations; (2) insights on the benefits and challenges of applying LLM-based techniques in high-risk contexts; and (3) lessons for integrating AI for sensemaking tasks in policing more broadly. We highlight that in high-stakes contexts, where accuracy and context-sensitivity are paramount, AI techniques must be balanced with other approaches and designed in close partnership with end-users.
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-15
Open AccessHuman-AI Interaction for Time-Critical Sensemaking in Missing Persons Investigations
Every year an estimated 200,000 people go missing in the UK alone. Missing persons investigations involve challenging time-critical sensemaking tasks based on fragmented data sources. This paper describes a mixed-methods participatory study evaluating data science and AI-driven techniques (summarisation, fact extraction, and data visualisation) for supporting these investigations as part of a human-centered workflow. A series of human-AI interfaces were iteratively designed and tested with search officers and domain experts at Police Scotland. Based on findings, we describe: (1) user and information needs for missing persons investigations; (2) insights on the benefits and challenges of applying LLM-based techniques in high-risk contexts; and (3) lessons for integrating AI for sensemaking tasks in policing more broadly. We highlight that in high-stakes contexts, where accuracy and context-sensitivity are paramount, AI techniques must be balanced with other approaches and designed in close partnership with end-users.
- Open Access
- 9 November 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-12
Shared Control for Text Editing in Virtual Reality with Voice, Gaze and Touch
- 9 November 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-12
Open AccessShared Control for Text Editing in Virtual Reality with Voice, Gaze and Touch
- Open Access
- 9 November 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-11
Probabilistic In-Plane Detection for Mid-Air Virtual Surface Interactions
- 9 November 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-11
Open AccessProbabilistic In-Plane Detection for Mid-Air Virtual Surface Interactions
- 6 October 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 32 (2) , 2007-2019
EnVisionVR: A Scene Interpretation Tool for Visual Accessibility in Virtual Reality
Effective visual accessibility in Virtual Reality (VR) is crucial for Blind and Low Vision (BLV) users. However, designing visual accessibility systems is challenging due to the complexity of 3D VR environments and the need for techniques that can be easily retrofitted into existing applications. While prior work has studied how to enhance or translate visual information, the advancement of Vision Language Models (VLMs) provides an exciting opportunity to advance the scene interpretation capability of current systems. This paper presents EnVisionVR, an accessibility tool for VR scene interpretation. Through a formative study of usability barriers, we confirmed the lack of visual accessibility features as a key barrier for BLV users of VR content and applications. In response, we used our findings from the formative study to inform the design and development of EnVisionVR, a novel visual accessibility system leveraging a VLM, voice input and multimodal feedback for scene interpretation and virtual object interaction in VR. An evaluation with 12 BLV users demonstrated that EnVisionVR significantly improved their ability to locate virtual objects, effectively supporting scene understanding and object interaction.
- 6 October 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 32 (2) , 2007-2019
EnVisionVR: A Scene Interpretation Tool for Visual Accessibility in Virtual Reality
Effective visual accessibility in Virtual Reality (VR) is crucial for Blind and Low Vision (BLV) users. However, designing visual accessibility systems is challenging due to the complexity of 3D VR environments and the need for techniques that can be easily retrofitted into existing applications. While prior work has studied how to enhance or translate visual information, the advancement of Vision Language Models (VLMs) provides an exciting opportunity to advance the scene interpretation capability of current systems. This paper presents EnVisionVR, an accessibility tool for VR scene interpretation. Through a formative study of usability barriers, we confirmed the lack of visual accessibility features as a key barrier for BLV users of VR content and applications. In response, we used our findings from the formative study to inform the design and development of EnVisionVR, a novel visual accessibility system leveraging a VLM, voice input and multimodal feedback for scene interpretation and virtual object interaction in VR. An evaluation with 12 BLV users demonstrated that EnVisionVR significantly improved their ability to locate virtual objects, effectively supporting scene understanding and object interaction.
- Open Access
ReachVox: Clutter-free Reachability Visualization for Robot Motion Planning in Virtual Reality
Human-Robot-Collaboration can enhance workflows by leveraging the mutual strengths of human operators and robots. Planning and understanding robot movements remain major challenges in this domain. This problem is prevalent in dynamic environments that might need constant robot motion path adaptation. In this paper, we investigate whether a minimalistic encoding of the reachability of a point near an object of interest, which we call ReachVox, can aid the collaboration between a remote operator and a robotic arm in VR. Through a user study (n=20), we indicate the strength of the visualization relative to a point-based reachability check-up.
Open AccessReachVox: Clutter-free Reachability Visualization for Robot Motion Planning in Virtual Reality
Human-Robot-Collaboration can enhance workflows by leveraging the mutual strengths of human operators and robots. Planning and understanding robot movements remain major challenges in this domain. This problem is prevalent in dynamic environments that might need constant robot motion path adaptation. In this paper, we investigate whether a minimalistic encoding of the reachability of a point near an object of interest, which we call ReachVox, can aid the collaboration between a remote operator and a robotic arm in VR. Through a user study (n=20), we indicate the strength of the visualization relative to a point-based reachability check-up.
- 17 July 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 31 (10) , 9104-9123
Working in Extended Reality in the Wild: Worker and Bystander Experiences of XR Virtual Displays in Public Real-World Settings
Although access to sufficient screen space is crucial to knowledge work, workers often find themselves with limited access to display infrastructure in remote or public settings. While virtual displays can be used to extend the available screen space through extended reality (XR) head-worn displays (HWD), we must better understand the implications of working with them in public settings from both users’ and bystanders’ viewpoints. To this end, we conducted two user studies. We first explored the usage of a hybrid AR display across real-world settings and tasks. We focused on how users take advantage of virtual displays and what social and environmental factors impact their usage of the system. A second study investigated the differences between working with a laptop, an AR system, or a VR system in public. We focused on a single location and participants performed a predefined task to enable direct comparisons between the conditions while also gathering data from bystanders. The combined results suggest a positive acceptance of XR technology in public settings and show that virtual displays can be used to accompany existing devices. We highlighted some environmental and social factors. We saw that previous XR experience and personality can influence how people perceive the use of XR in public. In addition, we confirmed that using XR in public still makes users stand out and that bystanders are curious about the devices, yet have no clear understanding of how they can be used.
- 17 July 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 31 (10) , 9104-9123
Working in Extended Reality in the Wild: Worker and Bystander Experiences of XR Virtual Displays in Public Real-World Settings
Although access to sufficient screen space is crucial to knowledge work, workers often find themselves with limited access to display infrastructure in remote or public settings. While virtual displays can be used to extend the available screen space through extended reality (XR) head-worn displays (HWD), we must better understand the implications of working with them in public settings from both users’ and bystanders’ viewpoints. To this end, we conducted two user studies. We first explored the usage of a hybrid AR display across real-world settings and tasks. We focused on how users take advantage of virtual displays and what social and environmental factors impact their usage of the system. A second study investigated the differences between working with a laptop, an AR system, or a VR system in public. We focused on a single location and participants performed a predefined task to enable direct comparisons between the conditions while also gathering data from bystanders. The combined results suggest a positive acceptance of XR technology in public settings and show that virtual displays can be used to accompany existing devices. We highlighted some environmental and social factors. We saw that previous XR experience and personality can influence how people perceive the use of XR in public. In addition, we confirmed that using XR in public still makes users stand out and that bystanders are curious about the devices, yet have no clear understanding of how they can be used.
- Open Access
- 7 July 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-19
Design Activity Simulation: Opportunities and Challenges in Using Multiple Communicative AI Agents to Tackle Design Problems
- 7 July 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-19
Open AccessDesign Activity Simulation: Opportunities and Challenges in Using Multiple Communicative AI Agents to Tackle Design Problems
- Open Access
- 25 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-15
Exclusion Rates among Disabled and Older Users of Virtual and Augmented Reality
- 25 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-15
Open AccessExclusion Rates among Disabled and Older Users of Virtual and Augmented Reality
- Open Access
- 25 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-15
Seeing and Touching the Air: Unraveling Eye-Hand Coordination in Mid-Air Gesture Typing for Mixed Reality
- 25 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-15
Open AccessSeeing and Touching the Air: Unraveling Eye-Hand Coordination in Mid-Air Gesture Typing for Mixed Reality