Jie Guo

Since micro-video is typically captured in complex real-world environments, it often contains a large amount of noise and redundant information. They can weaken the expression of key semantics, thereby affecting the model's accuracy in scene recognition and classification. Owing to the production, editing, or other influencing factors of micro-video content, the correlations among different modalities exhibit uncertainty, and the semantic strength of each modality is often inconsistent and affected by noise. These issues pose challenges to multimodal fusion, including uncertainties in the fusion weights and noise levels of each modality, as well as semantic differences across different modalities. To address the above problems, this paper proposes a Micro-video Scene Classification Method based on Generative Feature Disentanglement and Multimodal Dynamical Fusion (MDF-GFD). In the Generative Feature Disentanglement module, we tackle this problem from a data-generation perspective, decomposing each modality's data into modality‑specific, modality‑shared, and noise components. By minimizing the generative loss, we disentangle modality‑specific and modality‑shared representations from raw data and leverage them jointly for discriminative learning within individual modalities. In the Dynamic Multimodal Fusion module, the modality-specific and modality-shared components of each modality are integrated, and multimodal consistency and complementarity are effectively combined through an information-entropy-based dynamic fusion mechanism, which adaptively fuses modalities according to their information contribution. Experimental results demonstrate the effectiveness of the proposed MDF-GFD method in micro-video scene classification.

Variational Autoencoder(VAE) combines the ideas of autoencoders and variational inference, introducing the concept of latent space and variational inference to endow autoencoders to generate new images. VAE typically assumes that data follows a Gaussian distribution, but real data may follow other distributions. This inconsistency between the assumption and the true distribution can affect the modeling and reconstruction capabilities of VAE, which makes it difficult for traditional models to accurately capture the true distribution. To address the aforementioned issues, we propose a Prior Distribution Guided Gaussian Mixture Variational Autoencoder(PDGM-VAE). Specifically, we construct a Gaussian Mixture Prior Learner (GMPL) to capture complex features of the data distribution, enabling the model to learn and obtain a Gaussian mixture distribution that is reasonable and close to the real data distributions, which is then used as the prior distribution in the network. Furthermore, we build a Semantic-Aware Module with Embedded Prior Distribution (SAMEPD), integrating data and label information to learn the distribution parameters, enabling the network to learn and utilize the semantic knowledge contained in the labels. During training, by approximating the posterior distribution to the prior distribution, we enhance the model’s modeling and reconstruction capabilities, improving the quality of generated images. We evaluated the image generation task on five public datasets, and based on the FID metric, our proposed method outperformed other VAE methods.

In clinical practice, obtaining a large amount of labeled CNV data is very difficult. Semi-supervised learning can effectively utilize a large amount of unlabeled CNV data. Since CNV has complex features such as blurred and unevenly distributed pixels on the edges, there are differences in the segmentation difficulty between pixels in the same image. Existing semi-supervised segmentation methods do not consider the segmentation difficulty of pixels, which will reduce the segmentation accuracy. To address this problem, we propose a dual difficulty-aware adaptive pseudo-label learning (D2APL) method for semi-supervised CNV segmentation. The proposed dual difficulty awareness includes segmentation difficulty perception of pixels in labeled and unlabeled data. For labeled data, we propose a classification confidence-guided difficulty perception method. For unlabeled data, we propose a model stability-guided difficulty perception method. Finally, we propose a difficulty-aware self-training method to dynamically adjust the threshold of pseudolabels according to the difficulty, thereby improving the utilization of difficult-to-segment pixels in unlabeled data. Experimental results show that our method outperforms the state-of-the-art method in CNV segmentation.

Currently, surveillance cameras are extensively employed in public security, and vehicle re-identification has emerged as a burgeoning research area in computer vision. Nevertheless, vehicle re-identification grapples with the challenges of low intra-class similarity and high inter-class similarity. This study tackles these challenges by introducing a novel vehicle re-identification method that integrates convolution and vision transformer features. Specifically, channel-by-channel convolution is incorporated into the feedforward layer to bolster the extraction of local features. Concurrently, the information from the last layer’s class token and other patches is fused to yield a comprehensive and rich featured representation. Experiments conducted on the VeRi776 and VehicleID datasets validate that the proposed method outperforms current state-of-the-art vehicle re-identification methods.

Video semantic role grounding has gained substantial interest from both the academic and industrial communities. While existing methods have demonstrated considerable performance improvements, the influence of noisy and intra-object proposals, referring to proposals with the same object label, has yet to be explored in video semantic role grounding. In this study, we propose a semantic-aware contrastive learning network with proposal suppression to enhance the accuracy of grounding referenced objects. To fully exploit the semantic information in each semantic role, we introduce a novel semantic role encoding module that allows for precise representations of each semantic role. We also design a semantic-aware proposal suppression network to reduce the impact of noisy proposals on object representation learning. Additionally, we propose a proposal contrastive loss to improve cross-modal alignment and reduce the effect of irrelevant intra-object proposals. Extensive experiments on four datasets demonstrate that our model achieves significant improvements over state-of-the-art methods.

The development of the Internet has led to the emergence of micro-videos, and they have found favour with a large audience. Micro-video venue classification plays an important role in understanding its content. However, due to privacy protection and other reasons, some data is missing. The semantic strength of each modality is inconsistent in a microvideo, which brings challenges to the accuracy of its venue classification. To address these challenges, this study proposes a Micro-video Venue Classification method based on Adaptive Weighted Reconstruction (MVCAWR). The method complements missing data through reconstruction, and adaptively weights the reconstruction process by calculating the semantic similarity of the acoustic, textual, and visual modalities, respectively. The semantic information of the missing data modality, and the consistency and complementarity among them are preserved to the greatest extent. Experiments show that the proposed method is effective in the task of micro-video venue classification.