YMYuxin Mao

Yuxin Mao

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Yuxin Mao has a total of 19 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • LDT: Efficient Scalable Video Generation Using Linear Diffusion Transformer

    by Yuxin Mao, Zhen Qin, Xuyang Shen, Jinxing Zhou, Jing Zhang, Yiran Zhong, Yuchao Dai

    Diffusion-based models have established new bench marks in high-quality video generation. However, they face significant challenges regarding computational efficiency, particularly when dealing with long sequences, because of their inherent quadratic complexity. To address these challenges, we introduce the Linear Diffusion Transformer (LDT), which utilizes the linear attention mechanism to reduce computational complexity to a linear scale. This advancement facilitates the efficient generation of longer video sequences without sacrificing the quality of the output. Among the key innovations in our approach include two sophisticated positional encoding techniques: Exponential Relative Positional Encoding (ERPE) and 3D Linearized Relative Positional Encoding (3D-LRPE). By capturing local correlations between video tokens through these position-aware encodings, our model effectively preserves computational efficiency while maintaining the ability to model fine-grained spatiotemporal relationships. These techniques enhance the capability of the model to capture intricate spatial and temporal dependencies effectively. Furthermore, we propose the Spatial-Global Transformer (SGT), which alternates between spatial attention and global sequence attention to improve video modeling efficacy. Extensive experimental evaluations demonstrate that LDT not only achieves state of-the-art efficiency, but also excels in generation performance. In particular, in the “train short, test long” setting, our method exhibits remarkable scalability as the length of the video frames increases, underscoring its robustness for longer video sequences.

  • Cross-Modal Semantic Relevance is an Efficient Gatekeeper for Audio-Visual Video Parsing

    by Jinxing Zhou, Zhihui Li, Shijie Hao, Pengcheng Zhao, Yuxin Mao, Yanghao Zhou, Ruohao Guo, Yiran Zhong, Xiaojun Chang


  • Top Cited Papers

    Contrastive Conditional Latent Diffusion for Audio-Visual Segmentation

    by Yuxin Mao, Jing Zhang, Mochu Xiang, Yunqiu Lv, Dong Li, Yiran Zhong, Yuchao Dai

    Audio-visual Segmentation (AVS) is conceptualized as a conditional generation task, where audio is considered as the conditional variable for segmenting the sound producer(s). In this case, audio should be extensively explored to maximize its contribution for the final segmentation task. We propose a contrastive conditional latent diffusion model for audio-visual segmentation (AVS) to thoroughly investigate the impact of audio, where the correlation between audio and the final segmentation map is modeled to guarantee the strong correlation between them. To achieve semantic-correlated representation learning, our framework incorporates a latent diffusion model. The diffusion model learns the conditional generation process of the ground-truth segmentation map, resulting in ground-truth aware inference during the denoising process at the test stage. As our model is conditional, it is vital to ensure that the conditional variable contributes to the model output. We thus extensively model the contribution of the audio signal by minimizing the density ratio between the conditional probability of the multimodal data, e.g. conditioned on the audio-visual data, and that of the unimodal data, e.g. conditioned on the audio data only. In this way, our latent diffusion model via density ratio optimization explicitly maximizes the contribution of audio for AVS, which can then be achieved with contrastive learning as a constraint, where the diffusion part serves as the main objective to achieve maximum likelihood estimation, and the density ratio optimization part imposes the constraint. By adopting this latent diffusion model via contrastive learning, we effectively enhance the contribution of audio for AVS. The effectiveness of our solution is validated through experimental results on the benchmark dataset. Code and results are online via our project page: https://github.com/OpenNLPLab/DiffusionAVS.


  • Open Access

    TAVGBench: Benchmarking Text to Audible-Video Generation

    by Yuxin Mao, Xuyang Shen, Jing Zhang, Zhen Qin, Jinxing Zhou, Mochu Xiang, Yiran Zhong, Yuchao Dai

    The Text to Audible-Video Generation (TAVG) task involves generating videos with accompanying audio based on text descriptions. Achieving this requires skillful alignment of both audio and video elements. To support research in this field, we have developed a comprehensive Text to Audible-Video Generation Benchmark (TAVGBench), which contains over 1.7 million clips with a total duration of 11.8 thousand hours. We propose an automatic annotation pipeline to ensure each audible video has detailed descriptions for both its audio and video contents. We also introduce the Audio-Visual Harmoni score (AVHScore) to provide a quantitative measure of the alignment between the generated audio and video modalities. Additionally, we present a baseline model for TAVG called TAVDiffusion, which uses a two-stream latent diffusion model to provide a fundamental starting point for further research in this area. We achieve the alignment of audio and video by employing cross-attention and contrastive learning. Through extensive experiments and evaluations on TAVGBench, we demonstrate the effectiveness of our proposed model under both conventional metrics and our proposed metrics. The dataset and code can be found on this page https://npucvr.github.io/TAVGBench/ and on github https://github.com/OpenNLPLab/TAVGBench.


  • Generative Transformer for Accurate and Reliable Salient Object Detection

    by Yuxin Mao, Jing Zhang, Zhexiong Wan, Xinyu Tian, Aixuan Li, Yunqiu Lv, Yuchao Dai

    We explore the impact of transformers on accurate and reliable salient object detection. For accuracy, we integrate the transformer with a deterministic model and delineate its advantages in structural modeling. Regarding reliability, we address the transformer’s tendency to produce overly confident, incorrect predictions. To gauge reliability implicitly, we introduce a latent variable model within the transformer framework, termed the inferential generative adversarial network (iGAN). The stochastic nature of the latent variable facilitates the estimation of predictive uncertainty, which serves as an auxiliary measure of the model’s prediction reliability. Different from the conventional GAN, which defines the distribution of the latent variable as fixed standard normal distribution N (0, I). The proposed “iGAN” infers the latent variable by gradient-based Markov Chain Monte Carlo (MCMC), namely Langevin dynamics, leading to an input-dependent latent variable model. We apply our proposed iGAN to fully supervised salient object detection, explaining that iGAN within the transformer framework leads to both accurate and reliable salient object detection. The source code and experimental results are publicly available via our project page: https://npucvr.github.io/TransformerSOD.