Co-AttenDWG: Coattentive Dimension-Wise Gating and Expert Fusion for Multimodal Offensive Content Detection
- 20 October 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Artificial Intelligence
- Vol. 7 (5) , 2726-2738
Abstract
Multi-modal learning has emerged as a crucial research direction, as integrating textual and visual information can substantially enhance performance in tasks such as classification, retrieval, and scene understanding. Despite advances with large pre-trained models, existing approaches often suffer from insufficient cross-modal interactions and rigid fusion strategies, failing to fully harness the complementary strengths of different modalities. To address these limitations, we propose Co-AttenDWG, co-attention with dimension-wise gating, and expert fusion. Our approach first projects textual and visual features into a shared embedding space, where a dedicated co-attention mechanism enables simultaneous, fine-grained interactions between modalities. This is further strengthened by a dimension-wise gating network, which adaptively modulates feature contributions at the channel level to emphasize salient information. In parallel, dual-path encoders independently refine modality-specific representations, while an additional cross-attention layer aligns the modalities further. The resulting features are aggregated via an expert fusion module that integrates learned gating and self-attention, yielding a robust unified representation. Experimental results on the MIMIC and SemEval Memotion 1.0 datasets show that Co-AttenDWG achieves state-of-the-art performance and superior cross-modal alignment, highlighting its effectiveness for diverse multi-modal applications.Keywords
All Related Versions
Funding Information
- Bangladesh University of Business and Technology (BUBT) Research Fund
This publication has 32 references indexed in Scilit:
- Research on multimodal hate speech detection based on self-attention mechanism feature fusionThe Journal of Supercomputing, 2024
- Improved Baselines with Visual Instruction Tuning2024
- DJMF: A discriminative joint multi-task framework for multimodal sentiment analysis based on intra- and inter-task dynamicsExpert Systems with Applications, 2023
- Multimodal sentiment analysis based on fusion methods: A surveyInformation Fusion, 2023
- Joint multimodal sentiment analysis based on information relevanceInformation Processing & Management, 2022
- Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directionsInformation Fusion, 2022
- Relation-Aware Graph Attention Network for Visual Question Answering2019
- Image–text sentiment analysis via deep multimodal attentive fusionKnowledge-Based Systems, 2019
- Squeeze-and-Excitation Networks2018
- Deep Residual Learning for Image Recognition2016