Shanmin Wang

Multimodal Depression Estimation (MDE) aims to infer individual depression scores by analyzing various signals, such as visual, auditory, and language signals etc. Compared to Multimodal Depression Detection (MDD) methods that only provide discrete labels, MDE can provide a more refined score evaluation. However, symptom heterogeneity leads to differences in external behaviors among patients with similar depressive states, which limits the performance of direct regression MDE methods. To address this issue, we propose a Combined Depression Level and Deviation (CDLD) method for MDE, which separates samples at different depression levels and further analyzes subtle deviations within same level to improve estimation performance. Specifically, the Multilevel Depression Separation module constructs depression levels with inherent commonalities based on psychological theories and models the ordinality of these levels, thereby separating samples with different levels of depression. Building on this, the Level-specific Deviation Regression module contrasts sample features relative to level-specific anchors, regressing the subtle depression deviation. Finally, the depression level and deviation are integrated to infer the depression score more accurately. Experiments on the DAIC-WOZ, CMDC, SEARCH, and AVEC 2014 datasets demonstrate that the proposed coarse-to-fine method effectively mitigates the impact of symptom heterogeneity on depression estimation performance, showing significant advantages in MDE tasks. The code is publicly available at https://github.com/LIU70KG/CDLD.

Unsupervised Cross-Domain Facial Expression Recognition (CD-FER) aims to transfer the recognition ability from annotated source domains to unlabeled target domains. Despite the advancements in CD-FER techniques based on marginal distribution matching, certain inherent properties of facial expressions, such as implicit class margins and imbalanced class distributions, still leave room for improvement in existing models. In this paper, we propose a Class-Adaptive Self-Training (CAST) model for unsupervised CD-FER. In addition to domain alignment, the CAST model leverages self-training to learn pseudo labels and dually enhance aligned representations for explicit class distinction, considering implicit class margins. Furthermore, the CAST model conducts a comprehensive analysis of the negative effects of class distributions on pseudo-label learning from perspectives of class-level representation distributions and predicted probabilities, and subsequently proposes specific solutions. By jointly matching class-level representation distributions and class distributions, the CAST model successfully alleviates conditional distribution discrepancies between domains, which is particularly pertinent for facial expression properties. Experimental results, including assessments on multiple target domains and evaluations of multiple FER models, demonstrate the effectiveness, superiority, and universality of the CAST model. The code can be available at https://github.com/smwanghhh/CAST.

Disentangling facial expressions from other disturbing facial attributes in face images is an essential topic for facial expression recognition. Previous methods only care about facial expression disentanglement (FED) itself, ignoring the negative effects of other facial attributes. Due to the annotations on limited facial attributes, it is difficult for existing FED solutions to disentangle all disturbance from the input face. To solve this issue, we propose an expression complementary disentanglement network (ECDNet). ECDNet proposes to finish the FED task during a face reconstruction process, so as to address all facial attributes during disentanglement. Different from traditional re-construction models, ECDNet reconstructs face images by progressively generating and combining facial appearance and matching geometry. It designs the expression incentive (EIE) and expression inhibition (EIN) mechanisms, inducing the model to characterize the disentangled expression and complementary parts precisely. Facial geometry and appearance, generated in the reconstructed process, are dealt with to represent facial expressions and complementary parts, respectively. The combination of distinctive reconstruction model, EIE, and EIN mechanisms ensures the completeness and exactness of the FED task. Experimental results on RAF-DB, AffectNet, and CAER-S datasets have proven the effectiveness and superiority of ECDNet.

In this paper, we propose a new Multimodal Representation Learning (MRL) method for Multimodal Sentiment Analysis (MSA), which facilitates the adaptive interaction between modalities through Cooperative Sentiment Agents, named Co-SA. Co-SA comprises two critical components: the Sentiment Agents Establishment (SAE) phase and the Sentiment Agents Cooperation (SAC) phase. During the SAE phase, each sentiment agent deals with an unimodal signal and highlights explicit dynamic sentiment variations within the modality via the Modality-Sentiment Disentanglement (MSD) and Deep Phase Space Reconstruction (DPSR) modules. Subsequently, in the SAC phase, Co-SA meticulously designs task-specific interaction mechanisms for sentiment agents so that coordinating multimodal signals to learn the joint representation. Specifically, Co-SA equips an independent policy model for each sentiment agent that captures significant properties within the modality. These policies are optimized mutually through the unified reward adaptive to downstream tasks. Benefitting from the rewarding mechanism, Co-SA transcends the limitation of pre-defined fusion modes and adaptively captures unimodal properties for MRL in the multimodal interaction setting. To demonstrate the effectiveness of Co-SA, we apply it to address Multimodal Sentiment Analysis (MSA) and Multimodal Emotion Recognition (MER) tasks. Our comprehensive experimental results demonstrate that Co-SA excels at discovering diverse cross-modal features, encompassing both common and complementary aspects. The code can be available at https://github.com/smwanghhh/Co-SA.

Change detection (CD) is one of the most important methods for monitoring land surface changes. Recently, transformer-based models have been employed to CD. However, most of them focus on modeling the correlation within each image, while cannot well model the difference between bi-temporal images. In this paper, we propose a difference transformer network (DiFormer) to address this issue. Specifically, we propose a Token Exchange-based Difference Evaluation (TEDE) module to generate the inconsistency between the changed region and the surrounding context to highlight the difference between the bi-temporal images. In addition, to obtain semantically rich exchangeable tokens, we design a Multi-scale Semantic Perception (MSP) module, which provides assistance for difference modeling. In order to test the performance of DiFormer, we conduct qualitative and quantitative experiments on two public datasets, including LEVIR-CD and S2Looking. Experimental results show that our proposed DiFormer is able to achieve better results than several state-of-the-art models, with F1 of 92.15% on the LEVIR-CD dataset and 66.31% on the S2Looking dataset.

Facial Expression Recognition (FER) suffers from misrecognition due to the similarities between expressions. To address this issue, popular works replace original annotations with soft labels to reflect expression similarities. However, existing soft label learning (SLL) modules are independent of FER modules. In this paper, inspired by automatic control theory, we propose a bias-based soft label learning network for FER named EC-Net. For optimizing FER and SLL modules jointly, EC-Net constitutes the closed-loop feedback between the two modules by designing a module measuring and transmitting the bias between FER module predictions and target labels. Specifically, EC-Net contains three modules: E-subNet, C-subNet, and L-Transmitter. Firstly, E-subNet, i.e., the FER module, attempts to converge to target labels under the supervision of soft labels, acting as the executor. Then, L-Transmitter measures the bias between E-subNet predictions and target labels. It converts multiple discrete biases to the bias-based label through spectral clustering and transmits it to C-subNet. Finally, C-SubNet, i.e., the SLL module, generates soft labels from the bias-based label with a cascaded learner and progressively distinguishes similar expressions. It updates the learned soft labels for E-subNet, performing like the controller. Supervised by the bias-based soft label, E-subNet effectively reduces the dominant bias caused by similar expressions. We conduct extensive experiments on four popular benchmarks, demonstrating the effectiveness of applying closed-loop feedback in the FER task.

Facial expression recognition (FER) in the wild received broad concerns in which occlusion and pose variation are two key issues. This paper proposed a global multi-scale and local attention network (MA-Net) for FER in the wild. Specifically, the proposed network consists of three main components: a feature pre-extractor, a multi-scale module, and a local attention module. The feature pre-extractor is utilized to pre-extract middle-level features, the multi-scale module to fuse features with different receptive fields, which reduces the susceptibility of deeper convolution towards occlusion and variant pose, while the local attention module can guide the network to focus on local salient features, which releases the interference of occlusion and non-frontal pose problems on FER in the wild. Extensive experiments demonstrate that the proposed MA-Net achieves the state-of-the-art results on several in-the-wild FER benchmarks: CAER-S, AffectNet-7, AffectNet-8, RAFDB, and SFEW with accuracies of 88.42%, 64.53%, 60.29%, 88.40%, and 59.40% respectively. The codes and training logs are publicly available at https://github.com/zengqunzhao/MA-Net.

Automatic Dynamic Facial Expression Recognition (DFER) is challenging since how to effectively capture facial temporal dynamics is still an open problem. As the variations of facial expressions is a dynamic system that satisfies underlying rules, it is essential to explore the fundamental temporal properties for recognizing dynamic expressions. Inspired by the phase space reconstruction method for time series analysis, we propose a Phase Space Reconstruction Network (PSRNet) for learning spatio-temporal facial features. First, 3D convolutional neural networks are used to extract spatial and short-term features, which indicate each frame's state termed as observations in the phase space. All the observations compose the trajectory of the dynamical system. Then, a data-driven across-correlation matrix is inferred to reveal the relationship of the observations. With this matrix, the phase space reconstruction module reconstructs the trajectory by aggregating the observations adaptively. Reconstructed observations represent the gradual process of dynamic facial expressions, which is beneficial to recognize these expressions. The experiment results on Oulu, MMI, and CK+ demonstrate that PSRNet can extract more informative and representative spatio-temporal features for DFER. Moreover, the visualization reveals that the reconstructed features have global consistency in facial regions and find the underlying evolutionary pattern of dynamic facial expression.