Jörg Müller
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Jörg Müller has a total of 144 co-authors. Here are the top co-authors by number of shared publications:
- Albrecht Schmidt
LMU Munich
7 shared publications Antonio Krüger
University of Munster
6 shared publications- David Lindlbauer
Human-Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA
5 shared publications Robert Walter
Telekom Innovation Laboratories, TU Berlin
5 shared publicationsFlorian Alt
University of Stuttgart
4 shared publications- Tiare Feuchtner
HCI Group, University of Konstanz, Konstanz, Germany
4 shared publications
Latest Publications
- Open Access
- 21 September 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-7
From Brick to Click: Comparing LEGO Building in Virtual Reality and the Physical World
- 21 September 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-7
Open AccessFrom Brick to Click: Comparing LEGO Building in Virtual Reality and the Physical World
- 17 July 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 31 (10) , 9104-9123
Working in Extended Reality in the Wild: Worker and Bystander Experiences of XR Virtual Displays in Public Real-World Settings
Although access to sufficient screen space is crucial to knowledge work, workers often find themselves with limited access to display infrastructure in remote or public settings. While virtual displays can be used to extend the available screen space through extended reality (XR) head-worn displays (HWD), we must better understand the implications of working with them in public settings from both users’ and bystanders’ viewpoints. To this end, we conducted two user studies. We first explored the usage of a hybrid AR display across real-world settings and tasks. We focused on how users take advantage of virtual displays and what social and environmental factors impact their usage of the system. A second study investigated the differences between working with a laptop, an AR system, or a VR system in public. We focused on a single location and participants performed a predefined task to enable direct comparisons between the conditions while also gathering data from bystanders. The combined results suggest a positive acceptance of XR technology in public settings and show that virtual displays can be used to accompany existing devices. We highlighted some environmental and social factors. We saw that previous XR experience and personality can influence how people perceive the use of XR in public. In addition, we confirmed that using XR in public still makes users stand out and that bystanders are curious about the devices, yet have no clear understanding of how they can be used.
- 17 July 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 31 (10) , 9104-9123
Working in Extended Reality in the Wild: Worker and Bystander Experiences of XR Virtual Displays in Public Real-World Settings
Although access to sufficient screen space is crucial to knowledge work, workers often find themselves with limited access to display infrastructure in remote or public settings. While virtual displays can be used to extend the available screen space through extended reality (XR) head-worn displays (HWD), we must better understand the implications of working with them in public settings from both users’ and bystanders’ viewpoints. To this end, we conducted two user studies. We first explored the usage of a hybrid AR display across real-world settings and tasks. We focused on how users take advantage of virtual displays and what social and environmental factors impact their usage of the system. A second study investigated the differences between working with a laptop, an AR system, or a VR system in public. We focused on a single location and participants performed a predefined task to enable direct comparisons between the conditions while also gathering data from bystanders. The combined results suggest a positive acceptance of XR technology in public settings and show that virtual displays can be used to accompany existing devices. We highlighted some environmental and social factors. We saw that previous XR experience and personality can influence how people perceive the use of XR in public. In addition, we confirmed that using XR in public still makes users stand out and that bystanders are curious about the devices, yet have no clear understanding of how they can be used.
- Open Access
- 11 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-15
SIM2VR: Towards Automated Biomechanical Testing in VR
Automated biomechanical testing has great potential for the development of VR applications, as initial insights into user behaviour can be gained in silico early in the design process. In particular, it allows prediction of user movements and ergonomic variables, such as fatigue, prior to conducting user studies. However, there is a fundamental disconnect between simulators hosting state-of-the-art biomechanical user models and simulators used to develop and run VR applications. Existing user simulators often struggle to capture the intricacies of real-world VR applications, reducing ecological validity of user predictions. In this paper, we introduce sim2vr, a system that aligns user simulation with a given VR application by establishing a continuous closed loop between the two processes. This, for the first time, enables training simulated users directly in the same VR application that real users interact with. We demonstrate that sim2vr can predict differences in user performance, ergonomics and strategies in a fast-paced, dynamic arcade game. In order to expand the scope of automated biomechanical testing beyond simple visuomotor tasks, advances in cognitive models and reward function design will be needed.
- 11 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-15
Open AccessSIM2VR: Towards Automated Biomechanical Testing in VR
Automated biomechanical testing has great potential for the development of VR applications, as initial insights into user behaviour can be gained in silico early in the design process. In particular, it allows prediction of user movements and ergonomic variables, such as fatigue, prior to conducting user studies. However, there is a fundamental disconnect between simulators hosting state-of-the-art biomechanical user models and simulators used to develop and run VR applications. Existing user simulators often struggle to capture the intricacies of real-world VR applications, reducing ecological validity of user predictions. In this paper, we introduce sim2vr, a system that aligns user simulation with a given VR application by establishing a continuous closed loop between the two processes. This, for the first time, enables training simulated users directly in the same VR application that real users interact with. We demonstrate that sim2vr can predict differences in user performance, ergonomics and strategies in a fast-paced, dynamic arcade game. In order to expand the scope of automated biomechanical testing beyond simple visuomotor tasks, advances in cognitive models and reward function design will be needed.
- Open Access
Working in Extended Reality in the Wild: Worker and Bystander Experiences of XR Virtual Displays in Real-World Settings
Although access to sufficient screen space is crucial to knowledge work, workers often find themselves with limited access to display infrastructure in remote or public settings. While virtual displays can be used to extend the available screen space through extended reality (XR) head-worn displays (HWD), we must better understand the implications of working with them in public settings from both users' and bystanders' viewpoints. To this end, we conducted two user studies. We first explored the usage of a hybrid AR display across real-world settings and tasks. We focused on how users take advantage of virtual displays and what social and environmental factors impact their usage of the system. A second study investigated the differences between working with a laptop, an AR system, or a VR system in public. We focused on a single location and participants performed a predefined task to enable direct comparisons between the conditions while also gathering data from bystanders. The combined results suggest a positive acceptance of XR technology in public settings and show that virtual displays can be used to accompany existing devices. We highlighted some environmental and social factors. We saw that previous XR experience and personality can influence how people perceive the use of XR in public. In addition, we confirmed that using XR in public still makes users stand out and that bystanders are curious about the devices, yet have no clear understanding of how they can be used.
Open AccessWorking in Extended Reality in the Wild: Worker and Bystander Experiences of XR Virtual Displays in Real-World Settings
Although access to sufficient screen space is crucial to knowledge work, workers often find themselves with limited access to display infrastructure in remote or public settings. While virtual displays can be used to extend the available screen space through extended reality (XR) head-worn displays (HWD), we must better understand the implications of working with them in public settings from both users' and bystanders' viewpoints. To this end, we conducted two user studies. We first explored the usage of a hybrid AR display across real-world settings and tasks. We focused on how users take advantage of virtual displays and what social and environmental factors impact their usage of the system. A second study investigated the differences between working with a laptop, an AR system, or a VR system in public. We focused on a single location and participants performed a predefined task to enable direct comparisons between the conditions while also gathering data from bystanders. The combined results suggest a positive acceptance of XR technology in public settings and show that virtual displays can be used to accompany existing devices. We highlighted some environmental and social factors. We saw that previous XR experience and personality can influence how people perceive the use of XR in public. In addition, we confirmed that using XR in public still makes users stand out and that bystanders are curious about the devices, yet have no clear understanding of how they can be used.
- Open Access
- 12 June 2023
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Computer-Human Interaction
- Vol. 30 (3) , 1-50
Simulating Interaction Movements via Model Predictive Control
We present a Model Predictive Control (MPC) framework to simulate movement in interaction with computers, focusing on mid-air pointing as an example. Starting from understanding interaction from an Optimal Feedback Control (OFC) perspective, we assume that users aim at minimizing an internalized cost function, subject to the constraints imposed by the human body and the interactive system. Unlike previous approaches used in HCI, MPC can compute optimal controls for nonlinear systems. This allows to use state-of-the-art biomechanical models and handle nonlinearities that occur in almost any interactive system. Instead of torque actuation, our model employs second-order muscles acting directly at the joints. We compare three different cost functions and evaluate the simulation against user movements in a pointing study. Our results show that the combination of distance, control, and joint acceleration cost matches individual users’ movements best, and predicts movements with an accuracy that is within the between-user variance. To aid HCI researchers and designers in applying our approach for different users, interaction techniques, or tasks, we make our SimMPC framework, including CFAT, a tool to identify maximum voluntary torques in joint-actuated models, publicly available, and give step-by-step instructions.
- 12 June 2023
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Computer-Human Interaction
- Vol. 30 (3) , 1-50
Open AccessSimulating Interaction Movements via Model Predictive Control
We present a Model Predictive Control (MPC) framework to simulate movement in interaction with computers, focusing on mid-air pointing as an example. Starting from understanding interaction from an Optimal Feedback Control (OFC) perspective, we assume that users aim at minimizing an internalized cost function, subject to the constraints imposed by the human body and the interactive system. Unlike previous approaches used in HCI, MPC can compute optimal controls for nonlinear systems. This allows to use state-of-the-art biomechanical models and handle nonlinearities that occur in almost any interactive system. Instead of torque actuation, our model employs second-order muscles acting directly at the joints. We compare three different cost functions and evaluate the simulation against user movements in a pointing study. Our results show that the combination of distance, control, and joint acceleration cost matches individual users’ movements best, and predicts movements with an accuracy that is within the between-user variance. To aid HCI researchers and designers in applying our approach for different users, interaction techniques, or tasks, we make our SimMPC framework, including CFAT, a tool to identify maximum voluntary torques in joint-actuated models, publicly available, and give step-by-step instructions.
- Open Access
Towards a Deep(er) Understanding of Interaction through Modeling, Simulation, and Optimization
The traditional user-centered design process can hardly keep up with the ever faster technical development and increasingly diverse user preferences. As a solution, we propose to augment the tried-and-tested approach of conducting user studies with simulation and optimization of the entire human-computer interaction loop. This approach allows to better understand phenomena through explicit modeling, build virtual prototypes through simulation, and improve interaction techniques through optimization. Building predictive user models also supports the creation and validation of HCI theories, and constitutes a decisive step towards new, intelligent, and adaptive user interfaces. We report our experience in virtually developing new interaction techniques on the example of acoustic levitation, and present our optimization-based framework for HCI. With this, we strive to gain a better understanding of interaction and at the same time feed the discussion on questions such as which tools and tutorials are necessary to make virtual prototyping more accessible to different audiences.
Open AccessTowards a Deep(er) Understanding of Interaction through Modeling, Simulation, and Optimization
The traditional user-centered design process can hardly keep up with the ever faster technical development and increasingly diverse user preferences. As a solution, we propose to augment the tried-and-tested approach of conducting user studies with simulation and optimization of the entire human-computer interaction loop. This approach allows to better understand phenomena through explicit modeling, build virtual prototypes through simulation, and improve interaction techniques through optimization. Building predictive user models also supports the creation and validation of HCI theories, and constitutes a decisive step towards new, intelligent, and adaptive user interfaces. We report our experience in virtually developing new interaction techniques on the example of acoustic levitation, and present our optimization-based framework for HCI. With this, we strive to gain a better understanding of interaction and at the same time feed the discussion on questions such as which tools and tutorials are necessary to make virtual prototyping more accessible to different audiences.
- Open Access
- 26 October 2022
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 41 (5) , 1-14
OptiTrap: Optimal Trap Trajectories for Acoustic Levitation Displays
Acoustic levitation has recently demonstrated the ability to create volumetric content by trapping and quickly moving particles along reference paths to reveal shapes in mid-air. However, the problem of specifying physically feasible trap trajectories to display desired shapes remains unsolved. Even if only the final shape is of interest to the content creator, the trap trajectories need to determine where and when the traps need to be, for the particle to reveal the intended shape. We propose OptiTrap , the first structured numerical approach to compute trap trajectories for acoustic levitation displays. Our approach generates trap trajectories that are physically feasible and nearly time-optimal, and reveal generic mid-air shapes, given only a reference path (i.e., a shape with no time information). We provide a multi-dimensional model of the acoustic forces around a trap to model the trap-particle system dynamics and compute optimal trap trajectories by formulating and solving a non-linear path following problem. We formulate our approach and evaluate it, demonstrating how OptiTrap consistently produces feasible and nearly optimal paths, with increases in size, frequency, and accuracy of the shapes rendered, allowing us to demonstrate larger and more complex shapes than ever shown to date.
- 26 October 2022
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 41 (5) , 1-14
Open AccessOptiTrap: Optimal Trap Trajectories for Acoustic Levitation Displays
Acoustic levitation has recently demonstrated the ability to create volumetric content by trapping and quickly moving particles along reference paths to reveal shapes in mid-air. However, the problem of specifying physically feasible trap trajectories to display desired shapes remains unsolved. Even if only the final shape is of interest to the content creator, the trap trajectories need to determine where and when the traps need to be, for the particle to reveal the intended shape. We propose OptiTrap , the first structured numerical approach to compute trap trajectories for acoustic levitation displays. Our approach generates trap trajectories that are physically feasible and nearly time-optimal, and reveal generic mid-air shapes, given only a reference path (i.e., a shape with no time information). We provide a multi-dimensional model of the acoustic forces around a trap to model the trap-particle system dynamics and compute optimal trap trajectories by formulating and solving a non-linear path following problem. We formulate our approach and evaluate it, demonstrating how OptiTrap consistently produces feasible and nearly optimal paths, with increases in size, frequency, and accuracy of the shapes rendered, allowing us to demonstrate larger and more complex shapes than ever shown to date.
- Open Access
- 19 March 2021
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Vol. 5 (1) , 1-16
Orbuculum - Predicting When Users Intend to Leave Large Public Displays
We present a system, predicting the point in time when users of a public display are about to leave. The ability to react to users' intention to leave is valuable for researchers and practitioners alike: users can be presented additional content with the goal to maximize interaction times; they can be offered a discount coupon for redemption in a nearby store hence enabling new business models; or feedback can be collected from users right after they have finished interaction without interrupting their task. Our research consists of multiple steps: (1) We identified features that hint at users' intention to leave from observations and video logs. (2) We implemented a system capable of detecting such features from Microsoft Kinect's skeleton data and subsequently make a prediction. (3) We trained and deployed a prediction system with a Quiz game which reacts when users are about to leave (N=249), achieving an accuracy of 78%. The majority of users indeed reacted to the presented intervention.
- 19 March 2021
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Vol. 5 (1) , 1-16
Open AccessOrbuculum - Predicting When Users Intend to Leave Large Public Displays
We present a system, predicting the point in time when users of a public display are about to leave. The ability to react to users' intention to leave is valuable for researchers and practitioners alike: users can be presented additional content with the goal to maximize interaction times; they can be offered a discount coupon for redemption in a nearby store hence enabling new business models; or feedback can be collected from users right after they have finished interaction without interrupting their task. Our research consists of multiple steps: (1) We identified features that hint at users' intention to leave from observations and video logs. (2) We implemented a system capable of detecting such features from Microsoft Kinect's skeleton data and subsequently make a prediction. (3) We trained and deployed a prediction system with a Quiz game which reacts when users are about to leave (N=249), achieving an accuracy of 78%. The majority of users indeed reacted to the presented intervention.