Jiaolong Yang

Most current audio-driven facial animation research primarily focuses on generating videos with neutral emotions. While some studies have addressed the generation of facial videos driven by emotional audio, efficiently generating high-quality talking head videos that integrate both emotional expressions and style features remains a significant challenge. In this paper, we propose ESGaussianFace, an innovative framework for emotional and stylized audio-driven facial animation. Our approach leverages 3D Gaussian Splatting to reconstruct 3D scenes and render videos, ensuring efficient generation of 3D consistent results. We propose an emotion-audio-guided spatial attention method that effectively integrates emotion features with audio content features. Through emotion-guided attention, the model is able to reconstruct facial details across different emotional states more accurately. To achieve emotional and stylized deformations of the 3D Gaussian points through emotion and style features, we introduce two 3D Gaussian deformation predictors. Futhermore, we propose a multi-stage training strategy, enabling the step-by-step learning of the character’s lip movements, emotional variations, and style features. Our generated results exhibit high efficiency, high quality, and 3D consistency. Extensive experimental results demonstrate that our method outperforms existing state-of-the-art techniques in terms of lip movement accuracy, expression variation, and style feature expressiveness.

具身智能作为人工智能发展的关键领域，正面临数据异构性、强物理约束及交互昂贵等挑战，难以直接复制大语言模型的“大规模预训练+规模定律”范式。本文从数据、模型、系统与评测四个维度全面梳理了具身智能的前沿技术演进。在数据层面，文章提出了“数据金字塔”结构，主张利用底层庞大的仿真与互联网视频数据构建物理常识，通过中层人类交互数据进行行为映射，最终以顶层少量真机数据实现技能落地。在模型层面，探讨了主流视觉-语言-动作模型（VLA）的扩展瓶颈，并指出“世界模型”作为具身预训练的新方向，能够通过模拟环境动力学与未来预演，赋予智能体更强的物理直觉与泛化能力。在系统层面，观察到架构正从单一端到端模型向类操作系统的“分层架构”演进，实现高层语义规划与底层运动控制的解耦。最后，本文审视了当前评测体系在真实性与可复现性上的挑战，并对行走与操作一体化及具身智能“ImageNet时刻”的到来进行了展望。

Audio-driven 3D facial animation is crucial for enhancing the metaverse's realism, immersion, and interactivity. While most existing methods focus on generating highly realistic and lively 2D talking head videos by leveraging extensive 2D video datasets these approaches work in pixel space and are not easily adaptable to 3D environments. We present VASA-Rig, which has achieved a significant advancement in the realism of lip-audio synchronization, facial dynamics, and head movements. In particular, we introduce a novel rig parameter-based emotional talking face dataset and propose the Latents2Rig model, which facilitates the transformation of 2D facial animations into 3D. Unlike mesh-based models, VASA-Rig outputs rig parameters, instantiated in this paper as 174 Metahuman rig parameters, making it more suitable for integration into industry-standard pipelines. Extensive experimental results demonstrate that our approach significantly outperforms existing state-of-the-art methods in terms of both realism and accuracy.