Design and Implementation of the Transparent, Interpretable, and Multimodal (TIM) AR Personal Assistant
Authors:
- Erin McGowanEMErin McGowan
- Joao RulffJRJoao Rulff
- Sonia CasteloSCSonia Castelo
- Guande WuGWGuande Wu
- Shaoyu ChenSCShaoyu Chen
- Roque LopezRLRoque Lopez
- Bea SteersBSBea Steers
- Iran R. RomanIRIran R. Roman
- Fábio F. DiasFDFábio F. Dias
- Jing QianJQJing Qian
- Parikshit SolunkePSParikshit Solunke
- Michael MiddletonMMMichael Middleton
- 10 March 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Computer Graphics and Applications
- Vol. 45 (1) , 28-42
Abstract
The concept of an AI assistant for task guidance is rapidly shifting from a science fiction staple to an impending reality. Such a system is inherently complex, requiring models for perceptual grounding, attention, and reasoning, an intuitive interface that adapts to the performer's needs, and the orchestration of data streams from many sensors. Moreover, all data acquired by the system must be readily available for post-hoc analysis to enable developers to understand performer behavior and quickly detect failures. We introduce TIM, the first end-to-end AI-enabled task guidance system in augmented reality which is capable of detecting both the user and scene as well as providing adaptable, just-in-time feedback. We discuss the system challenges and propose design solutions. We also demonstrate how TIM adapts to domain applications with varying needs, highlighting how the system components can be customized for each scenario.Keywords
All Related Versions
Funding Information
- Defense Advanced Research Projects Agency (HR001121S0015)
This publication has 42 references indexed in Scilit:
- Detecting Twenty-Thousand Classes Using Image-Level Supervision2022
- Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications2022
- AR/MR Remote Collaboration on Physical Tasks: A ReviewRobotics and Computer-Integrated Manufacturing, 2021
- Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100International Journal of Computer Vision, 2021
- AI4AR: An AI-Based Mobile Application for the Automatic Generation of AR Contents2020
- Imaging and fusing time series for wearable sensor-based human activity recognitionInformation Fusion, 2020
- Evaluating the effectiveness of spatial augmented reality in smart manufacturing: a solution for manual working stationsThe International Journal of Advanced Manufacturing Technology, 2017
- State of science: mental workload in ergonomicsErgonomics, 2014
- Augmented reality on large screen for interactive maintenance instructionsComputers in Industry, 2014
- Working memory and prefrontal cortexNeuroscience Research, 1994