Yunyan Zhang

Given coupled sentence image pairs, Multimodal Aspect-based Sentiment Analysis (MABSA) aims to detect aspect terms and predict their sentiment polarity. While existing methods have made great efforts in aligning images and text for improved MABSA performance, they still struggle to effectively mitigate the challenge of the noisy correspondence problem (NCP): the text description is often not well-aligned with the visual content. To alleviate NCP, in this paper, we introduce Aspect-driven Alignment and Refinement (ADAR), which is a two-stage coarse-to-fine alignment framework. In the first stage, ADAR devises a novel Coarse-to-fine Aspect-driven Alignment Module, which introduces Optimal Transport (OT) to learn the coarse-grained alignment between visual and textual features. Then the adaptive filter bin is applied to remove the irrelevant image regions at a fine-grained level; In the second stage, ADAR introduces an Aspect-driven Refinement Module to further refine the cross-modality feature representation. Extensive experiments on two benchmark datasets demonstrate the superiority of our model over state-of-the-art performance in the MABSA task.

Multi-modal intent detection (MID) aims to comprehend users' intentions through diverse modalities, which has received widespread attention in dialogue systems. Despite the promising advancements in complex fusion mechanisms or architecture designs, challenges remain due to: (1) various noise and redundancy in both visual and audio modalities and (2) long-tailed distributions of intent categories. In this paper, to tackle the above two issues, we propose InMu-Net, a simple yet effective framework for MID from the Information bottleneck and Multi-sensory processing perspective. Our contributions lie in three aspects. First, we devise a denoising bottleneck module to filter out the intent-irrelevant information in the fused feature; Second, we introduce a saliency preservation loss to prevent the dropping of intent-relevant information; Ultimately, kurtosis regulation is introduced to maintain representation smoothness during the filtering process, mitigating the adverse impact of the long tail distribution. Comprehensive experiments on two MID benchmark datasets demonstrate the effectiveness of InMu-Net and its vital components. Impressively, a series of analyses reveal our denoising potential and robustness in low-resource, modality corruption, cross-architecture and cross-task scenarios.

Yuejia Xiang, Yunyan Zhang, Xiaoming Shi, Bo Liu, Wandi Xu, Xi Chen. Proceedings of the Fifteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-15). 2021.

In recent years, convolutional neural networks (CNNs) have shown great success in the scene classification of computer vision images. Although these CNNs can achieve excellent classification accuracy, the discriminative ability of feature representations extracted from CNNs is still limited in distinguishing more complex remote sensing images. Therefore, we propose a unified feature fusion framework based on attention mechanism in this paper, which is called Deep Discriminative Representation Learning with Attention Map (DDRL-AM). Firstly, by applying Gradient-weighted Class Activation Mapping (Grad-CAM) algorithm, attention maps associated with the predicted results are generated in order to make CNNs focus on the most salient parts of the image. Secondly, a spatial feature transformer (SFT) is designed to extract discriminative features from attention maps. Then an innovative two-channel CNN architecture is proposed by the fusion of features extracted from attention maps and the RGB (red green blue) stream. A new objective function that considers both center and cross-entropy loss are optimized to decrease the influence of inter-class dispersion and within-class variance. In order to show its effectiveness in classifying remote sensing images, the proposed DDRL-AM method is evaluated on four public benchmark datasets. The experimental results demonstrate the competitive scene classification performance of the DDRL-AM approach. Moreover, the visualization of features extracted by the proposed DDRL-AM method can prove that the discriminative ability of features has been increased.

Event argument extraction, which aims to identify arguments of specific events and label their roles, is a challenging subtask of event extraction. Previous approaches solve this problem in a two-stage manner that first extracts named entities as argument candidates and then determines their roles. However, many nested entities may be missed or wrongly predicted during the argument candidate extraction procedure, which substantially affects the performance of the downstream classifier. In this paper, we propose a novel one-step question answering based framework, which performs argument candidate extraction and argument role classification simultaneously to mitigate the error propagation problem in conventional two-stage methods. Since the conventional question answering based framework cannot be applied directly to this task, we design a Question Answering based Sequence Labeling (QA-SL) model to tackle inexistent argument roles and multiple argument token spans. Moreover, considering the overwhelming number of parameters in question answering based neural network models and the relatively small size of event extraction corpus, we fine-tune the pre-trained model from BERT to mitigate the data scarcity problem. Extensive experiments demonstrate the benefits of the proposed method, leading to a competitive performance compared with state-of-the-art methods. To the best of our knowledge, this is the first work to cast event argument extraction as a question answering task.

Network embedding (NE) aims to represent network information appropriately by learning low-dimensional and dense vectors for the nodes and edges of information network. Actually, the real world is almost full of heterogeneous information networks, which stimulates the emergence of heterogeneous information networks (HINs) embedding models. However, parts of existing HIN embedding models like meta-path-based methods only capture limited and aggregated information of relations, whereas some models based on metric or distance learning are usually of high computational complexity and slow training speed. In this paper, we present a novel heterogeneous information network embedding model, which applies dynamic projection metrics and translation mechanisms to the complicated heterogeneous information networks including multiple nodes and different relations. In order to overcome the imbalance of the distribution of relations in HIN and optimize the training process, we introduce an adaptive loss function for model optimization. Further more, we propose a hybrid model with baseline method as the initialization of the model. Experiments have been implemented on some real-world HIN datasets. And empirical results show that our model significantly outperforms the state-of-the-art representation learning models.

Siamese trackers learn the appearance model of the target in the first frame and then exploit the model to locate the target in the subsequent frames. Meanwhile, the appearance model remains unchanged in the subsequent frames. Due to the powerful feature extraction capability of the deep convolutional neural networks, Siamese trackers achieve advanced performance. However, due to the non-update of the appearance model and the changing appearance of the target, the problem of tracking drift occurs frequently, especially in the background clutters scenarios. In order to tackle this issue, we propose a motion model and a discriminative model. Firstly, the motion model of the target is constructed to determine whether the tracking drift occurs or not since the position of the target predicted by the motion model is smooth in timing but the position of the target predicted by the Siamese tracker may be not smooth. In this case, the temporal information is utilized to supplement the Siamese tracker which only employs the spatial information. Secondly, the discriminative model is learned to determine the final position of the target when the tracking drift happens. Finally, a flexible model update strategy of the discriminative model is presented. In order to demonstrate the generality of the proposed method, we apply it for two famous Siamese trackers, SiamFC and SiamRPN_DW. Extensive experiments on OTB2013, OTB2015, VOT2016, VOT2019 and GOT-10k benchmarks demonstrate that the proposed trackers outperform the baseline trackers and achieve the state-of-the-art performance, especially in the background clutters scenarios. To the best of our knowledge, we are the first time to propose motion guided Siamese trackers. Moreover, We can release our code to encourage more researches in this direction.