Sicheng Zhao

Multimodal emotion recognition (MER) is a cornerstone task in affective computing, aimed at discerning emotional clues from multimodal content. Recently, the emergence of multimodal large language models (MLLMs) has catalyzed a shift from MER to explainable multimodal emotion recognition (EMER), which promises not only accurate emotion prediction but also explainable reasoning. Despite notable progress achieved by reinforcement learning (RL) based methods, significant challenges persist in EMER. Existing models often produce lengthy, diffuse reasoning chains that obscure key spatiotemporal cues, and prevailing evaluation protocols offer only coarse-grained assessments of explainability. In response, we propose a two-component solution, each targeting a specific challenge. On the one hand, we introduce EmotionReasoner, a RL-based model optimization framework that integrates an explanation-aware reward and an emotion-polarity-oriented loss weighting strategy. The proposed reward jointly evaluates structure compliance, outcome correctness, and explanation fidelity, while the weighting strategy adaptively amplifies emotionally salient and polarity-critical tokens during loss computation. Collectively, the framework facilitates emotion-centric optimization toward more faithful, transparent, and explainable reasoning. On the other hand, we draw inspiration from related literature and augment existing evaluation protocols with three complementary metrics—MEER-E, LACE-E, and UTAI-E—that assess explanation fidelity, clarity, and practical utility, thereby providing a more comprehensive and fine-grained evaluation of EMER systems. Through extensive experiments on nine representative datasets, we demonstrate that the proposed framework outperforms existing ones in MER. Further evaluations also confirm its superiority in EMER, along with qualitative analysis that highlights the effectiveness and rationale behind the proposed evaluation metrics. Together, these contributions establish a unified pathway toward RL-driven EMER systems with comprehensive multimodal evaluation.

Multimodal large language models (MLLMs) have achieved remarkable success in multimodal applications. Their reliance on large-scale, internet-sourced data raises significant concerns regarding security and privacy. These concerns are especially critical when MLLM-based systems are deployed in agentic edge networks, where limited computational resources make retraining impractical for addressing privacy and security issues. Machine Unlearning (MU) has emerged as a critical technique for dealing with these issues. It enables the selective removal of targeted information from pre-trained models without costly retraining. However, the evaluation of MU for MLLMs in agentic edge networks remains inadequate. Existing benchmarks focus narrowly on entities and overlook unlearning of broader visual concepts and their semantic couplings. To bridge this gap, we introduce IEBench. This benchmark enables thorough assessment of MU for MLLM-based systems in agentic edge networks. IEBench features a fictitious dataset of identity entities and corresponding event scenes. It supports the evaluation of unlearning across these distinct yet entangled concepts. We use IEBench to evaluate eight MU methods and reveal their individual strengths and weaknesses. Our findings show that unlearning one concept can unintentionally degrade performance on related concepts within the same image. We term this challenge cross concept interference. Furthermore, we demonstrate the difficulty of unlearning identity and event concepts simultaneously. We also explore potential methods to mitigate these conflicting objectives.

Existing studies mostly decompose the document-level event extraction task into several sub-tasks, namely entity extraction, event type detection, and multi-event extraction under recognized event types. These sub-tasks are then executed independently in a step-by-step manner. This induces certain error propagation. Meanwhile, the associated semantics of entities functioning as arguments in multiple events have not been fully exploited, and the path expansion strategy relies heavily on the predefined role order. To address these issues, this paper proposes a novel strategy for joint document-level event extraction. First, inspired by the path expansion strategy, we devise a Role-Pair Event Tracing Graph (RP-ETG) to achieve the multi-event extraction sub-task. In this graph, tokens serve as nodes, with pairs designated as the forward edge types and tracing identifiers as the backward edge types. The forward edges serve to identify the roles that token nodes play in an event, while the backward edges are used to track which token nodes belong to the same event. Next, to determine the event type, this paper sequentially assigns numbers to the role-pair labels across different event types, ensuring that the labels for each event type fall into a distinct numerical interval. Thus, RP-ETG not only enables the accurate decoding of all events within a document, but also directly clarifies the argument roles that tokens assume in specific events under given event types. It can also capture the associated semantics of tokens when they act as arguments across multiple events. Leveraging this graph structure, we develop a graph-enhanced model for joint document-level event extraction. We evaluate the efficacy of our proposed method through exhaustive experiments on two public datasets. The results manifest its superiority over state-of-the-art baselines, validating the significant advancements enabled by our approach.

Recent advancements in autonomous driving, augmented reality, robotics, and embodied intelligence have necessitated 3D perception algorithms. However, current 3D perception methods, especially specialized small models, exhibit poor generalization in open scenarios. On the other hand, multimodal large language models (MLLMs) excel in general capacity but underperform in 3D tasks, due to weak 3D local spatial object perception, poor text-based geometric numerical output, and inability to handle camera focal variations. To address these challenges, we develop LLMI3D, and propose the following solutions: Spatial-Enhanced Local Feature Mining for better 3D spatial feature extraction, 3D Query Token-Derived Info Decoding for precise geometric regression, and Geometry Projection-Based 3D Reasoning for handling camera focal length variations. We are the first to adapt an MLLM for image-based 3D perception. Additionally, we have constructed the IG3D dataset, which provides fine-grained descriptions and question-answer annotations. Extensive experiments demonstrate that our LLMI3D achieves state-of-the-art performance, outperforming other methods by a large margin. We will publicly release our code, models, and dataset.

Emotional Video Captioning (EVC) seeks to generate video descriptions that are both factually accurate and emotionally expressive. However, existing approaches often lack structured semantic grounding and fine-grained temporal modeling, leading to incomplete or emotionally inconsistent captions. To address these issues, we propose HEART (Hierarchical Emotion-Aligned Representation with Temporal structure), a unified framework that jointly models hierarchical visual semantics and multi-scale temporal context. Specifically, HEART introduces a Hierarchical Semantic Extraction Module that decomposes visual content into entity-, action-, and event-level representations, providing a rich foundation for multi-level emotional alignment. A Temporal Pyramid Module captures short- and long-range temporal dependencies through multi-scale convolution, enabling temporally coherent captioning. Together, these components enable HEART to generate captions that are both emotionally grounded and temporally complete. To support this framework, we construct EmoStruct, a new benchmark dataset with fine-grained emotional annotations at the subject and predicate levels. Experiments on EmoStruct and public datasets demonstrate that HEART significantly outperforms prior methods in both semantic and emotional dimensions.