Zhibin Su

Diverse costume and decorative patterns in Dunhuang murals have significant historical and cultural value. However, natural erosion and material aging have caused cracks, flaking, and missing regions. Existing image restoration methods are still challenged by complex decorative patterns, including insufficient modeling of long-range structural dependencies, incoherent local textures, and weak global consistency. To address these problems, this paper proposes MuralsNet, an EdgeConnect-based inpainting framework adapted to the structural characteristics of Dunhuang decorative murals. The original two-stage strategy of EdgeConnect (G1, G2) is retained to provide explicit edge guidance, while two targeted modifications are introduced into the G2 image completion network. First, BiFormer is embedded into the bottleneck stage of G2 U-Net to capture global contextual information and long-range dependencies through a bi-level routing attention mechanism. Meanwhile, a Global–Local Fusion Module (GLM) is constructed in the decoder stage to facilitate the effective fusion of high-level semantic features and local detail features, thereby improving structural coherence and texture restoration in damaged regions. Experiments show that MuralsNet achieves the highest PSNR, SSIM, and Masked Edge F1, as well as the lowest LPIPS, among the evaluated methods, while LaMa obtains the lowest MAE. Visual comparisons further show that MuralsNet better preserves fine decorative contours and local structural details in several representative examples. Ablation experiments further show that BiFormer and the GLM provide complementary improvements within the proposed framework. These results demonstrate the potential of the proposed task-oriented integration strategy for the digital restoration of complex decorative mural images.

The composition of soundtracks for videos is a highly integrated process that synthesizes artistic creativity with technical expertise. It involves the precise synchronization of musical elements with the visual component to effectively enhance the artistic quality of the video. Herein, we aimed to generate harmonious and temporally aligned music based on video content, assisting artists in soundtrack composition. Existing approaches predominantly generate music based on human motion data from fixed-length videos, limiting their applicability. To overcome this limitation, we propose an autoregressive unit diffusion model, $\mathbf {W^{2}}$, for variable-length video-to-music generation. $\mathbf {W^{2}}$ segments videos and music into unified temporal units and employs a diffusion model to generate music units conditioned on structured video information and contextual music information. It then autoregressively generates coherent and consistent music sequences. This model employs structured video condition information, obtained through global encoding, to effectively generate music units that consistently align with the video content. At the same time, contextual music condition information is used to ensure the generation of a continuous sequence of music units matching the duration of the video. Unlike prior models relying on human skeletal features, $\mathbf {W^{2}}$ can be applied to not only dance videos but also free-style videos. Experimental results show that $\mathbf {W^{2}}$ generates music that is temporally synchronized and semantically aligned with visual content.

Ancient murals face significant degradation due to environmental and human factors. Traditional restoration is labor-intensive and inconsistent, while recent advancements in deep learning, particularly diffusion models, offer promising solutions. Diffusion models are known for their strong performance in generative tasks and are particularly well-suited for image restoration by iteratively refining degraded areas. These models can simulate the gradual addition and removal of noise, mimicking the natural restoration process, thereby enabling more precise reconstruction of damaged regions. Leveraging these advantages, this paper proposes a novel mural image restoration model architecture based on diffusion, aimed at addressing the specific challenges of restoring ancient murals. The proposed model combines a heterogeneous UNet structure with two key modules: pixel space augmentation block (PSAB) for enhancing spatial details and dual channel attention block (DCAB) for refining channel information. Experimental results show the model’s effectiveness in restoring large-scale damage and fine textures, outperforming existing methods in PSNR, SSIM, and LPIPS metrics.

This article investigates the impact of visual color perception on fine-grained emotion prediction in videos, analyzing the contribution of color perception features in fine-grained emotion prediction. A total of 20 subjects were involved in this experiment. First, 10 subjects conducted a fine-grained emotional subjective evaluation experiment on 50 video clips. Then, another 10 subjects conducted a subjective evaluation experiment of color perception annotation for these 50 video clips. On this basis, the correlation and mechanism between color perceptual features and fine-grained emotions were analyzed. Finally, a fine-grained emotion prediction model was established based on the combination of objective features and color perceptual features. It was also observed that, compared to using only objective features, incorporating perceptual features enhanced the model’s accuracy. This article also compared the importance of different perceptual features for each emotion and explained the mechanism between color perception and fine-grained emotions. By selecting the top 24 important features to predict emotions, the association between perceptual features and emotions was more effectively captured.

Most existing intelligent editing tools for music and video rely on the cross-modal matching technology of the affective consistency or the similarity of feature representations. However, these methods are not fully applicable to complex audiovisual matching scenarios, resulting in low matching accuracy and suboptimal audience perceptual effects due to ambiguous matching rules and associated factors. To address these limitations, this paper focuses on both the similarity and integration of affective distribution for the artistic audiovisual works of movie and television video and music. Based on the rich emotional perception elements, we propose a hybrid matching model based on feature canonical correlation analysis (CCA) and fine-grained affective similarity. The model refines KCCA fusion features by analyzing both matched and unmatched music–video pairs. Subsequently, the model employs XGBoost to predict relevance and to compute similarity by considering fine-grained affective semantic distance as well as affective factor distance. Ultimately, the matching prediction values are obtained through weight allocation. Experimental results on a self-built dataset demonstrate that the proposed affective matching model balances feature parameters and affective semantic cognitions, yielding relatively high prediction accuracy and better subjective experience of audiovisual association. This paper is crucial for exploring the affective association mechanisms of audiovisual objects from a sensory perspective and improving related intelligent tools, thereby offering a novel technical approach to retrieval and matching in music–video editing.

Film and TV video scenes contain rich art and design elements such as light and shadow, color, composition, and complex affects. To recognize the fine-grained affects of the art carrier, this paper proposes a multitask affective value prediction model based on an attention mechanism. After comparing the characteristics of different models, a multitask prediction framework based on the improved progressive layered extraction (PLE) architecture (multi-headed attention and factor correlation-based PLE), incorporating a multi-headed self-attention mechanism and correlation analysis of affective factors, is constructed. Both the dynamic and static features of a video are chosen as fusion input, while the regression of fine-grained affects and classification of whether a character exists in a video are designed as different training tasks. Considering the correlation between different affects, we propose a loss function based on association constraints, which effectively solves the problem of training balance within tasks. Experimental results on a self-built video dataset show that the algorithm can give full play to the complementary advantages of different features and improve the accuracy of prediction, which is more suitable for fine-grained affect mining of film and TV scenes.

Audience sentiment analysis is a popular research task in Music Information Retrieval (MIR). In this work, we aim to lean the prediction model of fine-grained music emotion for intelligent editing and retrieval. To satisfy the properties of time-series of music, the RNN-based model of BILSTM was selected as the bone. In particular, the traditional features and deep features extracted from the CNN architecture of spectrogram was fused with attention mechanism to enhance the recognition ability for different emotions. The ablation experiment has proved the necessary of the multi-dimensional features. Finally, the contrasting experiments on recent algorithms demonstrate that our proposed method for emotion prediction is effective, which is also suitable for small data sets.

The purpose of the research on emotion classification of film and television (TV) scene images is to hope that the computer can simulate the audience's emotional perception to judge positive or negative emotional trends of the image. In this paper, from the perspective of emotional semantics, a method of the emotion classification about the film and TV scene image is proposed. Firstly, the film and TV scene image dataset were established and assessed based on the subjective evaluation experiment. Then the theory of psychology and photographic art was used to extract image emotional features. Finally, particle swarm optimization (PSO)-support vector machine (SVM) algorithm based on feature importance dimension reduction was used for building the classifier. The experimental results show that the image dataset initially established in this paper can be helpful to the subsequent study on emotion classification of the film and TV scene image, and the accuracy of the positive and negative emotion classifier on this dataset is 0.87, which conforms to audience's emotional perception of the film and TV scene image.

This paper had investigated the perception of visual comfort for the disparity gradient of stereoscopic 3D(S3D) image though subjective evaluation experiments. For the extraction of gradient features, the calculation of absolute disparity and relative disparity were all considered. The experiment result shown that the relative disparity gradient mainly affects the visual comfort under the circumstance of positive disparity, while the absolute disparity gradient would affect the viewing experience with negative disparity. As a whole, the higher both the disparity gradient was, the lower the rating score was. The research achievement is of practical value, which can not only provide importance experimental support for the visual perception mechanism, but also offer the valid basis for feature extraction of stereoscopic images.