Multimodal Generative Semantic Communication Based on Latent Diffusion Model
Authors:
- 22 September 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-6
Abstract
In emergencies, the ability to quickly and accurately gather environmental data and command information, and to make timely decisions, is particularly critical. Traditional semantic communication frameworks, primarily based on a single modality, are susceptible to complex environments and lighting conditions, thereby limiting decision accuracy. To this end, this paper introduces a multimodal generative semantic communication framework named mm-GESCO. The framework ingests streams of visible and infrared modal image data, generates fused semantic segmentation maps, and transmits them using a combination of one-hot encoding and zlib compression techniques to enhance data transmission efficiency. At the receiving end, the framework can reconstruct the original multimodal images based on the semantic maps. Additionally, a latent diffusion model based on contrastive learning is designed to align different modal data within the latent space, allowing mm-GESCO to reconstruct latent features of any modality presented at the input. Experimental results demonstrate that mm-GESCO achieves a compression ratio of up to 200 times, surpassing the performance of existing semantic communication frameworks and exhibiting excellent performance in downstream tasks such as object classification and detection.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- A Unified Multi-Task Semantic Communication System for Multimodal DataIEEE Transactions on Communications, 2024
- Generative Joint Source-Channel Coding for Semantic Image TransmissionIEEE Journal on Selected Areas in Communications, 2023
- Task-Oriented Multi-User Semantic CommunicationsIEEE Journal on Selected Areas in Communications, 2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Task-Oriented Multi-User Semantic Communications for VQAIEEE Wireless Communications Letters, 2021
- Semantic Communication Systems for Speech TransmissionIEEE Journal on Selected Areas in Communications, 2021
- VAE for Joint Source-Channel Coding of Distributed Gaussian Sources over AWGN MAC2020
- Deep Joint Source-Channel Coding for Wireless Image TransmissionIEEE Transactions on Cognitive Communications and Networking, 2019
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes2017