A Design of Smart Beaker Structure and Interaction Paradigm Based on Multimodal Fusion Understanding
Open Access
- 18 September 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 8, 173766-173778
Abstract
Virtual chemistry experiment is an important teaching tool in middle schools. It can not only help students understand the experimental principles, but also help them memorize the experimental procedures. However, there are still some problems in virtual chemistry experiments. First, in existing research, user intentions are often misunderstood and cannot be accurately understood. Second, the existing research fail to identify the user’s wrong actions, which reduce the accuracy of the experiment. Third, the user sense of operation and realism is not strong during the experiment, which reduces the user’s experience. Finally, the lack of navigation guidance for experimental operations increases user learning time. In order to solve these problems. This paper proposes a scheme for establishing an intelligent navigational chemical laboratory based on multimodal fusion. First of all, we design a new smart beaker structure with perceptual ability, which can be used to complete most chemical experiments and give users a real sense of experience. Besides, we propose a multimodal fusion understanding algorithm, which reduces the misidentification of the experiment and better understands the real intention of the user. Finally, intelligent navigation and wrong behavior recognition functions are added to the experimental equipment, which improves the efficiency of human-computer interaction. The results show that Compared with the existing virtual laboratory or system, the chemical laboratory scheme proposed in this paper through multimodal fusion understanding algorithm greatly reduces the user’s memory load and improves the success rate of the experiment. Moreover, through the combination of virtual and real, the virtual chemistry experiment not only improves the authenticity of the operation, but also stimulates the students’ interest in learning, which is well received by users.Keywords
Funding Information
- National Key R&D Program of China (2018YFB1004901)
- Independent Innovation Team Project of Jinan City (2019GXRC013)
This publication has 26 references indexed in Scilit:
- Exploring Preservice Teachers’ Emotional Experiences in an Immersive Virtual Teaching Simulation through Facial Expression RecognitionInternational Journal of Human–Computer Interaction, 2018
- Move to Smart Learning Environment: Exploratory Research of Challenges in Computer Laboratory and Design Intelligent Virtual Laboratory for eLearning TechnologyEURASIA Journal of Mathematics, Science and Technology Education, 2018
- Augmented and Virtual Reality2017
- Multimodal Fusion With Reference: Searching for Joint Neuromarkers of Working Memory Deficits in SchizophreniaIEEE Transactions on Medical Imaging, 2017
- CHEMOTION: A gesture based chemistry virtual laboratory with leap motionComputer Applications in Engineering Education, 2017
- Teaching with a Dual-Channel Classroom Feedback System in the Digital Classroom EnvironmentIEEE Transactions on Learning Technologies, 2016
- Virtual Chime-Bells Experimental System Based on Multi-modal Fusion2015
- Multimodal Multi-Channel On-Line Speaker Diarization Using Sensor Fusion Through SVMIEEE Transactions on Multimedia, 2015
- Multimodal Biometric Human Recognition for Perceptual Human–Computer InteractionIEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews), 2010
- Multimodal fusion for multimedia analysis: a surveyMultimedia Systems, 2010