A Timestamp-Independent Haptic–Visual Synchronization Method for Haptic-Based Interaction System
Open Access
Abstract
The booming haptic data significantly improve the users’ immersion during multimedia interaction. As a result, the study of a Haptic-based Interaction System has attracted the attention of the multimedia community. To construct such a system, a challenging task is the synchronization of multiple sensorial signals that is critical to the user experience. Despite audio-visual synchronization efforts, there is still a lack of a haptic-aware multimedia synchronization model. In this work, we propose a timestamp-independent synchronization for haptic–visual signal transmission. First, we exploit the sequential correlations during delivery and playback of a haptic–visual communication system. Second, we develop a key sample extraction of haptic signals based on the force feedback characteristics and a key frame extraction of visual signals based on deep-object detection. Third, we combine the key samples and frames to synchronize the corresponding haptic–visual signals. Without timestamps in the signal flow, the proposed method is still effective and more robust in complicated network conditions. Subjective evaluation also shows a significant improvement of user experience with the proposed method.Keywords
Funding Information
- National Natural Science Foundation of China (62171134, JAT200024)
- Foundation for Middle-aged and Young Educational Committee of Fujian Province (62171134, JAT200024)
This publication has 28 references indexed in Scilit:
- Toward Haptic Communications Over the 5G Tactile InternetIEEE Communications Surveys & Tutorials, 2018
- Wearable Haptics and Immersive Virtual Reality Rehabilitation Training in Children With Neuromotor ImpairmentsIEEE Transactions on Neural Systems and Rehabilitation Engineering, 2018
- 3D Convolutional Neural Networks for Cross Audio-Visual Matching RecognitionIEEE Access, 2017
- Haptic Guidance in Dynamic Environments Using Optimal Reciprocal Collision AvoidanceIEEE Robotics and Automation Letters, 2017
- Analysing the concrete compressive strength using Pearson and Spearman2017
- Realizing the Tactile Internet: Haptic Communications over Next Generation 5G Cellular NetworksIEEE Wireless Communications, 2016
- Spatial Pyramid Pooling in Deep Convolutional Networks for Visual RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Influences of inter-stream synchronization errors among haptic media, sound, and video on quality of experience in networked ensemble2013
- Assessing the importance of audio/video synchronization for simultaneous translation of video sequencesMultimedia Systems, 2012
- An Efficient Audio-Video Synchronization Methodology2007