Context‐Aware Mixed Reality: A Learning‐Based Framework for Semantic‐Level Interaction
Open Access
- 14 November 2019
- journal article
- research article
- Published by Wiley in Computer Graphics Forum
- Vol. 39 (1) , 484-496
Abstract
Mixed reality (MR) is a powerful interactive technology for new types of user experience. We present a semantic‐based interactive MR framework that is beyond current geometry‐based approaches, offering a step change in generating high‐level context‐aware interactions. Our key insight is that by building semantic understanding in MR, we can develop a system that not only greatly enhances user experience through object‐specific behaviours, but also it paves the way for solving complex interaction design challenges. In this paper, our proposed framework generates semantic properties of the real‐world environment through a dense scene reconstruction and deep image understanding scheme. We demonstrate our approach by developing a material‐aware prototype system for context‐aware physical interactions between the real and virtual objects. Quantitative and qualitative evaluation results show that the framework delivers accurate and consistent semantic information in an interactive MR environment, providing effective real‐time semantic‐level interactions.Keywords
This publication has 36 references indexed in Scilit:
- ORB-SLAM: A Versatile and Accurate Monocular SLAM SystemIEEE Transactions on Robotics, 2015
- Interactive Urban Context-Aware Visualization via Multiple Disocclusion OperatorsIEEE Transactions on Visualization and Computer Graphics, 2015
- CAST: Effective and Efficient User Interaction for Context-Aware Selection in 3D Particle CloudsIEEE Transactions on Visualization and Computer Graphics, 2015
- Material recognition in the wild with the Materials in Context Database2015
- Caffe2014
- Dense 3D semantic mapping of indoor scenes from RGB-D images2014
- Robust real-time visual odometry for dense RGB-D mapping2013
- Semantic interaction for visual text analytics2012
- TextonBoost for Image Understanding: Multi-Class Object Recognition and Segmentation by Jointly Modeling Texture, Layout, and ContextInternational Journal of Computer Vision, 2007
- Random sample consensusCommunications of the ACM, 1981