Chunfang Yang

Multimodal representation learning aims to integrate heterogeneous modalities—such as text, image, audio, and video—into a unified representation. However, modality imbalance and inter-modal interference often lead to modality collapse, limiting generalization, especially under noisy or incomplete inputs. To address this, we propose Spiking-based Multimodal Representation Learning(S-MRL), a spiking neural network (SNN) framework that reformulates multimodal learning as a process of spike sequence encoding and integration. S-MRL employs biologically inspired spike encoders and a Spiking Response Model(SRM) to extract temporally structured, modality-specific features. A Gaussian Kernel Spike Distance(GKSD) is introduced to construct a shared latent space across modalities, while Joint Entropy(JE) captures cross-modal information and preserves semantic diversity. A Recurrent Spiking Neural Networks(SNN) architecture is further designed to mitigate forgetting and stabilize training. During inference, an informative fusion mechanism aggregates spike-based representations, and Cross-Entropy(CE) supervises downstream classification. Extensive experiments on four multimodal benchmarks— under both complete and missing modality settings—demonstrate that S-MRL achieves 4%–19% accuracy improvements over existing methods, while enhancing energy efficiency, robustness, and generalization. These results establish S-MRL as a principled and biologically plausible solution for robust multimodal representation learning.

Steganography is a technique for embedding secret messages into digital media while preserving perceptual integrity and avoiding detection. Robust steganography specifically enables secure communication through lossy channels with disturbances, where a critical challenge lies in the receiver's ability to accurately reconstruct hidden messages from corrupted stego data-a process termed error-correction. Recent advancements in stego coding schemes for robust steganography leverage probabilistic decoding to enhance message error-correction accuracy. These schemes employ Maximum-A-Posteriori (MAP) decoders that integrate channel knowledge, i.e. the probability distribution of stego symbols derived from received data, thus necessitating precise estimation of such channel knowledge. While existing approach for Channel Knowledge Estimation (CKE) relies on heuristic, handcrafted methods grounded in empirical assumptions, its effectiveness is limited by the inherent complexity of media content and the diversity of channel disturbance. Besides, it is not compatible with some of the embedding method for robust steganography. To address these limitations, we propose a deep learning-based framework for universal channel knowledge estimation, which significantly improves error-correction performance. Furthermore, we extend this framework to multi-channels robust steganography scenarios, formulating a novel multi-channels knowledge estimation paradigm to enhance message correctness through transmission in multiple channels. Experimental results on various state-of-the-art robust steganography methods demonstrate that our approach outperforms existing CKE methods and coding schemes.

Index selection is a crucial component in database query optimization. Traditional database index selection is inefficient when handling large-scale and complex structured query language (SQL) queries, and existing methods often overlook index maintenance costs and the necessity of updates. To address these issues, a network-optimized Monte Carlo tree search (NMCTS)-based index selection model, called tree-based index selection (Tree-IS), is proposed by using the sampling-based reinforcement learning algorithm Monte Carlo tree search (MCTS). This model compresses the action space through workload-driven query template extraction and candidate index (CI) generation techniques. It integrates a novel query state representor and an execution-plan-based index value model (IVM), accurately characterizing the database environment while providing reliable action criteria for index search. On this basis, Tree-IS further leverages a state abstraction network (SAN), a policy network, and a value network (VN) to optimize the search logic of MCTS, achieving the rapid identification of optimal index sets and a significant improvement in query efficiency. Extensive experiments are conducted on popular datasets, including join order benchmark (JOB), transaction processing performance council benchmark holistic (TPC-H), and transaction processing performance council benchmark decision support (TPC-DS), to evaluate the proposed method across multiple metrics. The results demonstrate that the quality of indices selected by the Tree-IS model is significantly superior to that of existing methods.

Traditional pedestrian trajectory prediction models focus on spatio–temporal data without proper consideration of individual interactions with the environment, mutual interactions, and contextual information, resulting in low prediction performance in real applications. In this article, we propose a new pedestrian trajectory prediction model called spatial social force graph neural network (SSF-GNN). First, SSF-GNN adopts a gate recurrent unit (GRU) network and a CenterNet network to capture pedestrian trajectory features and environmental features from historical trajectory sequences. Particularly, SSF-GNN can quantify pedestrian interactions and context-awareness information based on social force. Second, SSF-GNN employs a graph neural network to integrate social influence and hidden states of pedestrians. The distance between adjacent trajectory points is approximated by the weighted average summation of pedestrian historical trajectories. Third, SSF-GNN employs a new interaction function between pedestrians by considering the distance between pedestrians, as well as the movement speed of pedestrians in the social force model, to accurately predict trajectories of pedestrians. Extensive experiments are conducted on two famous datasets, and the results demonstrate SSF-GNN’s outperforms the state-of-the-art models, where average displacement error (ADE) is reduced by more than 25.6%, and final displacement error (FDE) is reduced by more than 15.4%. When predicting a pedestrian’s trajectory in the next eight frames of locations, SSF-GNN outperforms other models significantly with an accuracy of 69.71%.

Current Android apps rely heavily on external storage. When using the external storage, apps apply different security strategies (e.g., randomizing file name, encrypting file content) to prevent privacy risks. Even so, we find that privacy risks still exist, i.e., information leakage of apps' file operations. Users follow their habits to run apps and some of the apps conduct file operations on external storage, which potentially expose users' regular activities. In this paper, we conduct the first empirical study on this problem and implement a file-operation-based pipeline, OP-PERUSE. Besides a dataset of 5,359,339 file operation events collected from the volunteers, we crawl 22,484 app records from the third-party app statistics websites. By combining these data, we get some timely and fine-grained information about the users, e.g., current affiliation, position, habits, etc. To further understand this problem's severity, we conduct a static code analysis on 15,098 apps. We find that 1,305 (8.64%) apps tend to collect file operation events, and more than half of these apps adopt the third-party SDKs which gather file operation events, indicating this problem could have persisted over a long period of time. To prevent this problem, we provide some security recommendations for different stakeholders.

Various malicious activities performed by social bots have brought a crisis of trust to online social networks. Existing social bot detection methods often overlook the significance of community structure features and effective fusion strategies for multimodal features. To counter these limitations, we propose BotCF, a novel social bot detection method that incorporates community features and utilizes cross-attention fusion for multimodal features. In BotCF, we extract community features using a community division algorithm based on deep autoencoder-like non-negative matrix factorization. These features capture the social interactions and relationships within the network, providing valuable insights for bot detection. Furthermore, we employ cross-attention fusion to integrate the features of the account’s semantic content, properties, and community structure. This fusion strategy allows the model to learn the interdependencies between different modalities, leading to a more comprehensive representation of each account. Extensive experiments conducted on three publicly available benchmark datasets (Twibot20, Twibot22, and Cresci-2015) demonstrate the effectiveness of BotCF. Compared to state-of-the-art social bot detection models, BotCF achieves significant improvements in accuracy, with an average increase of 1.86%, 1.67%, and 0.47% on the respective datasets. The detection accuracy is boosted to 86.53%, 81.33%, and 98.21%, respectively.