Explainable Interfaces for Rapid Gaze-Based Interactions in Mixed Reality
Authors:
- Mengjie YuMYMengjie Yu
- Dustin HarrisDHDustin Harris
- Ian JonesIJIan Jones
- Ting ZhangTZTing Zhang
- Yue LiuYLYue Liu
- Naveen SendhilnathanNSNaveen Sendhilnathan
- Narine KokhlikyanNKNarine Kokhlikyan
- Fulton WangFWFulton Wang
- Co TranCTCo Tran
- Jordan L. LivingstonJLJordan L. Livingston
- Krista E. TaylorKTKrista E. Taylor
- Zhenhong HuZHZhenhong Hu
Preprint
Abstract
Gaze-based interactions offer a potential way for users to naturally engage with mixed reality (XR) interfaces. Black-box machine learning models enabled higher accuracy for gaze-based interactions. However, due to the black-box nature of the model, users might not be able to understand and effectively adapt their gaze behaviour to achieve high quality interaction. We posit that explainable AI (XAI) techniques can facilitate understanding of and interaction with gaze-based model-driven system in XR. To study this, we built a real-time, multi-level XAI interface for gaze-based interaction using a deep learning model, and evaluated it during a visual search task in XR. A between-subjects study revealed that participants who interacted with XAI made more accurate selections compared to those who did not use the XAI system (i.e., F1 score increase of 10.8%). Additionally, participants who used the XAI system adapted their gaze behavior over time to make more effective selections. These findings suggest that XAI can potentially be used to assist users in more effective collaboration with model-driven interactions in XR.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Motor “laziness” constrains fixation selection in real-world tasksProceedings of the National Academy of Sciences, 2024
- PalmGazer: Unimanual Eye-hand Menus in Augmented Reality2023
- Gaze as an Indicator of Input Recognition ErrorsProceedings of the ACM on Human-Computer Interaction, 2022
- Explaining Deep Neural Networks and Beyond: A Review of Methods and ApplicationsProceedings of the IEEE, 2021
- Neural correlates of goal-directed and non–goal-directed movementsProceedings of the National Academy of Sciences, 2021
- A comparison of instance-level counterfactual explanation algorithms for behavioral and textual data: SEDC, LIME-C and SHAP-CAdvances in Data Analysis and Classification, 2020
- The Effect of Task on Visual Attention in Interactive Virtual EnvironmentsACM Transactions on Applied Perception, 2019
- Explanation in artificial intelligence: Insights from the social sciencesArtificial Intelligence, 2019
- Gaze typing in virtual reality2018
- VRpursuits2018