QoE of cross-modally mapped Mulsemedia: an assessment using eye gaze and heart rate
Open Access
- 3 January 2020
- journal article
- research article
- Published by Springer Nature in Multimedia Tools and Applications
- Vol. 79 (11-12) , 7987-8009
Abstract
A great deal of research effort has been put in exploring crossmodal correspondences in the field of cognitive science which refer to the systematic associations frequently made between different sensory modalities (e.g. high pitch is matched with angular shapes). However, the possibilities cross-modality opens in the digital world have been relatively unexplored. Therefore, we consider that studying the plasticity and the effects of crossmodal correspondences in a mulsemedia setup can bring novel insights about improving the human-computer dialogue and experience. Mulsemedia refers to the combination of three or more senses to create immersive experiences. In our experiments, users were shown six video clips associated with certain visual features based on color, brightness, and shape. We examined if the pairing with crossmodal matching sound and the corresponding auto-generated haptic effect, and smell would lead to an enhanced user QoE. For this, we used an eye-tracking device as well as a heart rate monitor wristband to capture users’ eye gaze and heart rate whilst they were experiencing mulsemedia. After each video clip, we asked the users to complete an on-screen questionnaire with a set of questions related to smell, sound and haptic effects targeting their enjoyment and perception of the experiment. Accordingly, the eye gaze and heart rate results showed significant influence of the cross-modally mapped multisensorial effects on the users’ QoE. Our results highlight that when the olfactory content is crossmodally congruent with the visual content, the visual attention of the users seems shifted towards the correspondent visual feature. Crosmodally matched media is also shown to result in an enhanced QoE compared to a video only condition.Keywords
Funding Information
- Horizon 2020 Framework Programme (688503)
- CAPES (8881.187844/2018-01)
This publication has 66 references indexed in Scilit:
- Authoring multisensorial contentSignal Processing: Image Communication, 2013
- Smelling Shapes: Crossmodal Correspondences Between Odors and ShapesChemical Senses, 2012
- Crossmodal correspondences: A tutorial reviewAttention, Perception, & Psychophysics, 2011
- Cross-modal integration between odors and abstract symbolsNeuroscience Letters, 2010
- Implicit association between basic tastes and pitchNeuroscience Letters, 2009
- Temporal Frequency Channels Are Linked across Audition and TouchCurrent Biology, 2009
- Cross-Modal Associations Between Odors and ColorsChemical Senses, 2006
- The Effect of Visual Images on Perception of OdorsChemical Senses, 2005
- On cross-modal similarity: Auditory–visual interactions in speeded discrimination.Journal of Experimental Psychology: Human Perception and Performance, 1987
- Synesthesia in Children: Association of Colors with Pure Tone FrequenciesThe Journal of Genetic Psychology, 1956