Aimin Hao

Multi-modality motion stylization presents a solution to the challenge of generating flexible, stylized motion based on multimodal style inputs. Historically, motion stylization has grappled with the difficulty of balancing content and style, often prioritizing one at the expense of the other. This paper addresses the complex challenge of multi-modality content-style duality, achieving a sophisticated integration that both preserves and enhances the core narrative through nuanced stylistic modifications. We propose a Multi-modality Latent Diffusion Model (MM-LDM), a novel framework that leverages diffusion models under multi-modality conditions, including motion style (text-based or motion-based), motion content, and motion trajectory components. A central innovation in our approach is the introduction of a Multi-condition Denoiser, which carefully balances the preservation of primary content with the dynamic integration of style and trajectory as secondary conditions. This multi-modality guidance mechanism, implemented during the denoising process, ensures that new styles are seamlessly integrated with the original content. It gives rise to more authentic and cohesive motion stylization outcomes, establishing a new benchmark in computer animation. To further refine the control over the text-based motion style, we introduce an LLM parser that converts broad motion descriptions into detailed, part-specific representations. By decomposing the human body into movement-related parts, our method significantly enhances the precision and effectiveness of text-based motion stylization, enabling fine-grained control over individual body parts. Our model's effectiveness and generalization capabilities have been rigorously validated through extensive experiments, including text-based motion stylization and generating stylized motion with video sources, which all demonstrate the potential of our MM-LDM to advance the state-of-the-art motion stylization.

Generating continuous and expressive human motion from textual descriptions is a critical challenge in applications such as gaming and filmmaking. Existing methods often struggle to maintain global coherence, realistic frame continuity, and smooth transitions. To address these limitations, we propose FCMD, a novel diffusion-based model for generating cohesive motion sequences from fine-grained textual descriptions. FCMD introduces three key innovations: (1) Fine-grained Text Fusion, which integrates detailed textual cues with transitional narratives to enhance semantic consistency; (2) History Motion Guidance, ensuring motion accuracy and consistency across consecutive frames; and (3) Smooth Stitching Sampling, which leverages preceding and current motion information to achieve seamless transitions. Additionally, FCMD employs a large language model (LLM) to refine motion datasets by extracting fine-grained textual descriptions. Extensive experiments demonstrate that FCMD outperforms state-of-the-art methods in generating coherent, natural, and highly controllable motion sequences.

Speech-driven 3D facial animation has notable applications in the VR domain, including virtual anchors and digital avatars, etc. However, producing facial animations that convey complex emotional expressions remains a substantial challenge. Existing methods struggle to simultaneously achieve accurate lip synchronization, natural facial expressions, and realistic emotional representation. Significantly, the impact of head pose on boosting facial emotional expressiveness has not been thoroughly investigated. To address these issues, we propose EmoPoseFace, a novel Diffusion-based network to generate speech-driven 3D emotional facial animations with synchronized head poses. Our method employs a dual-branch conditional generation architecture to separately model facial expressions and head poses, integrating emotion and head-pose conditions for coherent facial expression-pose control. In addition, we design the Global-local Facial Fine-grained Editing Module (GL-FFE), which achieves emotional enhancement of facial expressions and fine-grained facial modification, while maintains the naturalness and authenticity of facial movements. Extensive experiments demonstrate that our approach outperforms existing methods in lip-sync accuracy and emotional detail preservation. The introduction of head pose control and GL-FFE significantly expands the expressiveness of emotional virtual facial animation, and the fine-grained editing is widely approved in perceptual user studies. The source code can be found at https://github.com/zaqwsx6/EmoPoseFace.

Generative content is increasingly applied in VR to provide immersive experiences, yet maintaining high generation quality remains challenging for audiovisual effects. Particularly in dynamic fluid phenomena, achieving realism and presence requires adherence to physical laws. To accomplish this objective, this work proposes an audiovisual synergy fluid animation generation framework, which enhances immersion by improving motion texture fidelity and audiovisual consistency. It comprises Detail-Enhanced Texture generator (DET) and Physics-Guided Audio generator (PGA). DET integrates Global-Local Physics guidance (GLP) and Temporal Texture Modeling (TTM) to produce video textures, explicitly optimizing dynamic details by leveraging local motion cues and assigned cumulative differences. PGA incorporates Visual Semantic Augmenter (VSA) and Rhythm Semantic Adapter (RSA) to synchronize audio by fusing static visual semantics with dynamic motion semantics to improve temporal coherence. By integrating DET and PGA, this framework strengthens audiovisual immersion in VR natural dynamic scenes from both visual and auditory perspectives. Quantitative and qualitative evaluations demonstrate that our approach surpasses most existing methods in terms of texture realism and audiovisual synchronization, offering new insights for advancing immersive experiences in dynamic VR phenomena.

Immersive VR applications demand high resolutions and refresh rates, posing significant challenges for real-time rendering. Foveated rendering mitigates this cost by exploiting properties of the Human Visual System (HVS), but conventional approaches often rely on oversimplified heuristic models that neglect high-level attentional cues, resulting in artifacts in peripheral regions. To this end, we present a neural saliency-driven foveated ray tracing framework that overcomes these limitations. Our method introduces a motion-aware foveation model to capture temporal dynamics and employs a lightweight convolutional neural network to predict saliency maps that reflect complex attentional patterns derived from eye-gaze data. The combination of these guides adaptive path tracing and filtering, enabling perceptually optimized rendering with minimal artifacts. Experimental results show that our approach improves perceptual quality over prior methods while sustaining real-time performance.