Sanglu Lu

Multimodal graph neural networks (MGNNs) have become an effective paradigm for modeling complex multimedia data where nodes are described by heterogeneous modalities. However, real-world multimodal graphs are often modality-incomplete, which limits the effectiveness of existing MGNNs. Meanwhile, most graph pre-training approaches assume fully observed modalities and fail to generalize well when modalities are missing. In this work, we study the problem of pre-training for incomplete multimodal graph learning, aiming to learn robust representations for modality-missing graphs. Therefore, we propose a novel and unified framework, Multimodal Graph Semantic Hub Induced Pre-training (MG-SHIP). We introduce a learnable cross-modal semantic hub graph that serves as a semantic mediator, explicitly routing information across modality-specific graphs. Through this hub-mediated interaction mechanism, the model captures transferable intra-modal knowledge and cross-modal relational dependencies. This design improves representation robustness under modality-missing conditions. To further stabilize pre-training, we incorporate a relational consistency objective to encourage robust cross-modal relations and a distribution-aware regularization to mitigate modality discrepancies. The pre-trained semantic hub further supports lightweight adaptation to downstream graphs with incomplete modalities. Extensive experiments on five real-world benchmark datasets demonstrate that MG-SHIP consistently outperforms state-of-the-art methods under different levels of label scarcity and modality incompleteness.

Time series analysis is a fundamental data mining task that has made encouraging progress in many real-world scenarios. Supervised training methods based on empirical risk minimization have proven their effectiveness on specific tasks and datasets. However, the acquisition of well-annotated data is costly, and a large amount of unlabeled series data is under-utilized. Due to distributional shifts across various domains and different patterns of interest across multiple tasks. The problem of cross-domain multi-task migration remains a significant challenge. To address these problems, this article proposes a novel cross-domain approach based on Wave Quantization for Time Series (termed as WQ4TS), which can be combined with any advanced time series model and applied to diverse downstream tasks. Specifically, we transfer the data from different domains into a common spectral latent space and enable the model to learn the temporal pattern knowledge of different domains directly from the common space and utilize it for the inference of downstream tasks, thereby mitigating the challenge of heterogeneous migration. The establishment of spectral latent space brings at least three benefits, cross-domain migration capability thus adapting to zero- and few-shot scenarios without relying on priori knowledge, general compatible cross-domain framework without changing the existing model structure, and robust modeling capability thus achieving SOTA results in multiple downstream tasks. To demonstrate the effectiveness of the proposed approach, we conduct extensive experiments including three important tasks: forecasting, imputation, and classification. And three common real-world scenarios are simulated: full-data, few-shot, and zero-shot. The proposed WQ4TS achieves the best performance on 87.5% of all tasks. Concretely, WQ4TS achieved 25.8% and 44.1% improvements in MSE metric for few-shot and zero-shot forecasting tasks, respectively, and demonstrated excellent 24.9% increase in average accuracy on few-shot classification tasks. The source codes of WQ4TS are publicly available on https://github.com/Xiang-Kai/WQ4TS .

Speech enhancement is crucial for reliable communication in noisy environments. However, the lack of a priori knowledge about target speech characteristics in conventional systems often leads to erroneous extraction of interfering speech as desired signals during noise suppression, significantly compromising system performance. Recently, researchers have proposed to use the side-channel signal as an assist to denoise noisy speech. This paper proposes BoneSE, a multimodal speech enhancement approach using bone conduction. The basic idea of BoneSE is to perceive the bone-conducted sound with an IMU sensor embedded in the Commercial Off-The-Shelf (COTS) earphone and then leverage the correlations between the bone conduction signal and audio signal for speech enhancement. However, the lack of high-frequency components in the IMU modality brings data imbalance and hinders data fusion. To address this challenge, we explore and model the relationship between multimodal signals and design a fusion module according to the time and frequency correlation. Moreover, to balance fast processing and denoising performance, we propose bone conduction-based noise level metrics to measure the noise level. To accommodate different noise levels, we propose an adaptive model selection approach based on reinforcement learning to select the proper denoising model, thereby optimizing the latency. Experiments on two datasets show that the proposed method performs favorably against state-of-the-art methods and can enhance speech in low SNR scenarios.

Using multiple Unmanned Aerial Vehicles (UAVs) in video surveillance greatly enhances real-time monitoring of large areas. However, images captured by UAVs are separate and limited in view, making stitching crucial for a comprehensive perspective. Current methods combine sensor and visual modalities for video stitching but face challenges in robustness and real-time performance. Environmental disturbances increase errors in sensor and visual data, reducing accuracy and stability, while single-frame stitching incurs high computational costs and slow speeds. To address these issues, we propose SkyEye , a real-time video stitching method for UAV surveillance systems based on multi-modal perception. Specifically, we design a mutual verification method to assess the quality of visual and inertial data. Frames with higher confidence are prioritized for precise stitching. To reduce computational costs, we employ a reference frame scheme that reuses the perspective transformation and feature points of the reference frame for subsequent frames. Meanwhile, we design a video stitching framework based on a per-frame parallel unit to speed up the algorithm execution. We have implemented a prototype of SkyEye and carried out extensive evaluations. Experiment results show that SkyEye outperforms state-of-the-art methods, improving speed by 66.1% and accuracy by 28.2%.