Jialiang Cheng
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Jialiang Cheng has a total of 5 co-authors. Here are the top co-authors by number of shared publications:
- Chao Sun
Wuhan National Laboratory for Optoelectronics, and Key Laboratory of Information Storage System, Engineering Research Center of Data Storage Systems and Technology, Ministry of Education of China, and School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China
1 shared publication - Chuanbo Zhu
Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, China
1 shared publication - Han Liang
Information Engineering College, Henan University of Science and Technology, Luoyang, China
1 shared publication - Jincai Chen
Huazhong University of Science and Technology
1 shared publication - Min Chen
South China University of Technology
1 shared publication
Latest Publications
- Open Access
- 26 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 261-270
SCLAV: Supervised Cross-modal Contrastive Learning for Audio-Visual Coding
Audio and vision are important senses for high-level cognition, and their special strong correlation makes audio-visual coding a crucial factor in many multimodal tasks. However, there are two challenges in audio-visual coding. First, the heterogeneity of multimodal data often leads to misalignment of cross-modal features under the same sample, which reduces their representation quality. Second, most self-supervised learning frameworks are constructed based on instance semantics, and the generated pseudo labels introduce additional classification noise. To address these challenges, we propose a Supervised Cross-modal Contrastive Learning Framework for Audio-Visual Coding (SCLAV). Our framework includes an audio-visual coding network composed of an inter-modal attention interaction module and an intra-modal self-integration module, which leverage multimodal complementary and hidden information for better representation. Additionally, we introduce a supervised cross-modal contrastive loss to minimize the distance between audio and vision features of the same instance, and use weak labels of multimodal data to eliminate the feature-oriented classification noise. Extensive experiments on the AVE and XD-Violence datasets demonstrate that SCLAV outperforms the state-of-the-art results, even with limited computational resources.
- 26 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 261-270
Open AccessSCLAV: Supervised Cross-modal Contrastive Learning for Audio-Visual Coding
Audio and vision are important senses for high-level cognition, and their special strong correlation makes audio-visual coding a crucial factor in many multimodal tasks. However, there are two challenges in audio-visual coding. First, the heterogeneity of multimodal data often leads to misalignment of cross-modal features under the same sample, which reduces their representation quality. Second, most self-supervised learning frameworks are constructed based on instance semantics, and the generated pseudo labels introduce additional classification noise. To address these challenges, we propose a Supervised Cross-modal Contrastive Learning Framework for Audio-Visual Coding (SCLAV). Our framework includes an audio-visual coding network composed of an inter-modal attention interaction module and an intra-modal self-integration module, which leverage multimodal complementary and hidden information for better representation. Additionally, we introduce a supervised cross-modal contrastive loss to minimize the distance between audio and vision features of the same instance, and use weak labels of multimodal data to eliminate the feature-oriented classification noise. Extensive experiments on the AVE and XD-Violence datasets demonstrate that SCLAV outperforms the state-of-the-art results, even with limited computational resources.