Cross-Modal Transmission Strategy
- 16 August 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 32 (6) , 3991-4003
Abstract
Multi-modal services, typically integrated by visual, audio, and haptic signals, have been considered as promising killer services in 5G and beyond 5G era. However, due to essential difference among these signals, how to guarantee quality of multi-modal services is a significant technical challenge. Existing transmission schemes, which deliver and process each modality signal separately, cannot meet such requirements as low latency, high reliability, and high throughput. To get over the dilemma, this paper proposes a general cross-modal transmission strategy by taking advantage of the potential correlation among modalities, which consists of a delivery mechanism at the sender and a signal restoration procedure at the receiver. On the one hand, a visual-aided haptic content compression method is designed for the delivery mechanism. By utilizing the category correlation among modalities, haptic signals with similar visual content can be effectively compressed, reducing transmission burden. On the other hand, a fine-grained haptic to image synthesis (FHIS) approach is proposed for realizing signal restoration. Through exploring strong matching properties among modalities, the FHIS can restore the impaired, missing, delayed visual images from the received haptic signals. Experiments on a standard visual-haptic database and a practical platform evaluate the performance of the constructed cross-modal transmission strategy.Keywords
Funding Information
- National Natural Science Foundation of China (62071254)
- Open Research Fund of the Key Laboratory of Broadband Wireless Communication and Sensor Network Technology, Nanjing University of Posts and Telecommunications (NJUPT), Ministry of Education (JZNY202111)
- Priority Academic Program Development of Jiangsu Higher Education Institutions
This publication has 37 references indexed in Scilit:
- Generalization of the Dark Channel Prior for Single Image RestorationIEEE Transactions on Image Processing, 2018
- TV: A Sparse Optimization Method for Impulse Noise Image RestorationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- A Multiplexing Scheme for Multimodal TeleoperationACM Transactions on Multimedia Computing, Communications, and Applications, 2017
- Video Streaming in Content-Centric Mobile Networks: Challenges and SolutionsIEEE Wireless Communications, 2017
- Multimodal Feature-Based Surface Material ClassificationIEEE Transactions on Haptics, 2016
- Rethinking the Inception Architecture for Computer Vision2016
- 5G-Enabled Tactile InternetIEEE Journal on Selected Areas in Communications, 2016
- SSIM-Based Coarse-Grain Scalable Video CodingIEEE Transactions on Broadcasting, 2015
- Toward Haptic Cinematography: Enhancing Movie Experiences with Camera-Based Haptic EffectsIEEE MultiMedia, 2014
- QoS-Based Cross-Layer Scheduling for Wireless Multimedia Transmissions with Adaptive Modulation and CodingIEEE Transactions on Communications, 2013