YLYaoyao Liu

Yaoyao Liu

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Yaoyao Liu has a total of 82 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Native Real-Time Interactive Omni Intelligence: A Unified Framework

    by Yifan Shen, Zhuoqing Zhong, Xinzhuo Li, Jiateng Liu, Bingxuan Li, Wei Cao, Pei Tian, Lin Zhu, Jian Xu, Zewei Chen, et al.

    The rapid progress of large language models has established strong foundations for reasoning, knowledge, and generation, while the integration of vision, audio, video, and other modalities has extended these capabilities toward increasingly unified intelligence across modalities, which we refer to as Omni Intelligence. Yet, real-world assistants must do more than process completed multimodal prompts. They must perceive evolving streams, respond while new evidence is arriving, handle hesitation, overlap, interruption, and changing intent, and coordinate reasoning or tool use that may proceed asynchronously. We refer to this broader capability as \( \textbf{Interactive Omni Intelligence} \): the ability to maintain and exercise omni intelligence under the temporal and adaptive demands of real-time interaction. The question guiding this paper is: How can foundation-model capabilities be carried into real-time interaction while preserving both intelligence and interactivity?We develop a conceptual framework connecting two development paths. One adapts pretrained language or omni models through streaming interfaces, modality modules, post-training, and orchestration. The other develops model-native systems that learn temporal structure, continuous multimodal processing, generation, and interaction control. Across both paths, we compare representations, architectures, fusion, data, training objectives, serving, and evaluation. Their comparison reveals an Intelligence--Interactivity Trade-off between preserving reasoning and knowledge, and meeting the timing, coordination, and resource demands of continuous operation. We examine Multi-Rate Interactive Intelligence as a cross-cutting direction, including foreground--background designs that coordinate fast interaction with slower reasoning, retrieval, memory, and tools. Finally, we identify open problems in evolving state, persistent and proactive interaction, joint evaluation, and safe open-world operation. The resulting framework connects capability preservation, interaction control, and multi-rate coordination.

  • Active View Selection with Perturbed Gaussian Ensemble for Tomographic Reconstruction

    by Yulun Wu, Ruyi Zha, Wei Cao, Yingying Li, Yuanhao Cai, Yaoyao Liu


  • How To Teach Large Multimodal Models New Skills?

    by Zhen Zhu, Yiming Gong, Yao Xiao, Yaoyao Liu, Derek Hoiem

  • AC3S: Adaptive Conditioning for 3D-Aware Synthetic Data Generation

    by Eric Ji, Qiran Hu, Wufei Ma, Sarthak Jain, Yingying Li, Minh N. Do, Yaoyao Liu

  • Open Access

    Motion2VecSets: Non-Rigid Shape Reconstruction and Tracking With 4D Latent Set Diffusion

    by Jiapeng Tang, Wei Cao, Biao Zhang, Chang Luo, Yaoyao Liu, Matthias Nießner

    We introduce Motion2VecSets, a 4D diffusion model for dynamic surface mesh generation from various ambiguous observations, including a sequence of RGB images, sparse and partial point clouds, and low-resolution voxel grids. While recent methods using neural field representations have shown success in modeling non-rigid objects, conventional feed-forward architectures struggle with noisy, partial, or sparse observations due to their deterministic nature. To address the inherent one-to-many mapping problem, we introduce a diffusion model that explicitly learns the shape and motion distribution of non-rigid objects through an iterative denoising process of compressed latent representations. The diffusion-based priors provide more plausible and diverse reconstructions under ambiguous conditions. Instead of relying on global latent codes, we represent 4D dynamics using latent sets. This novel 4D representation captures local shape and deformation patterns, leading to more accurate non-linear motion capture and significantly improving generalization capacity to unseen motions and identities. For temporally coherent tracking, we jointly denoise latent sets across frames and enable cross-frame information exchange. To reduce computational cost, we design an interleaved spatial-temporal attention block that alternately aggregates deformation latents along spatial and temporal dimensions. Extensive experiments on datasets of humans, animals, and articulated objects demonstrate that Motion2VecSets outperforms prior methods in reconstructing and tracking non-rigid deformations from various imperfect observations. Our implementation is available at https://vveicao.github.io/projects/Motion2VecSets/.


  • Top Cited Papers
    Open Access

    FreeOrbit4D: Training-Free Arbitrary Camera Redirection for Monocular Videos via Foreground-Complete 4D Reconstruction

    by Wei Cao, Hao Zhang, Fengrui Tian, Yulun Wu, Yingying Li, Shenlong Wang, Ning Yu, Yaoyao Liu

    Camera redirection aims to replay a dynamic scene from a single monocular video under a user-specified camera trajectory. However, large-angle redirection is inherently ill-posed: a monocular video captures only a narrow spatio-temporal view of a dynamic 3D scene, providing severely limited observations of the underlying 4D world. The key challenge is therefore to recover a complete and coherent plenoptic representation from this limited input, with consistent geometry and coherent motion. While recent diffusion-based methods achieve impressive visual generation quality, they often break down under large-angle viewpoint changes far from the original trajectory, where missing visual grounding leads to severe geometric ambiguity and temporal inconsistency. To address this, we present FreeOrbit4D, an effective training-free framework that tackles this geometric ambiguity by recovering a foreground-complete 4D proxy as structural grounding for video generation. We obtain this proxy by decoupling foreground and background reconstructions: we unproject the monocular video into a static background and partial foreground point clouds in a unified global space, then leverage an object-centric multi-view diffusion model to synthesize multi-view images and reconstruct complete foreground point clouds in canonical object space. By aligning the canonical foreground point cloud to the global scene space via dense pixel-synchronized 3D–3D correspondences and projecting the foreground-complete 4D proxy onto target camera viewpoints, we provide geometric scaffolds (e.g., depth/visibility cues) that guide a conditional video diffusion model. Extensive experiments show that FreeOrbit4D produces more faithful and temporally coherent redirected videos under challenging large-angle trajectories, and our foreground-complete 4D proxy further opens a potential avenue for practical applications such as edit propagation and 4D data generation. Project page: https://freeorbit4d.vision.ischool.illinois.edu/


  • iNeMo: Incremental Neural Mesh Models for Robust Class-Incremental Learning

    by Tom Fischer, Yaoyao Liu, Artur Jesslen, Noor Ahmed, Prakhar Kaushik, Angtian Wang, Alan L. Yuille, Adam Kortylewski, Eddy Ilg


  • Open Access

    Online Hyperparameter Optimization for Class-Incremental Learning

    by Yaoyao Liu, Yingying Li, Bernt Schiele, Qianru Sun

    Class-incremental learning (CIL) aims to train a classification model while the number of classes increases phase-by-phase. An inherent challenge of CIL is the stability-plasticity tradeoff, i.e., CIL models should keep stable to retain old knowledge and keep plastic to absorb new knowledge. However, none of the existing CIL models can achieve the optimal tradeoff in different data-receiving settings—where typically the training-from-half (TFH) setting needs more stability, but the training-from-scratch (TFS) needs more plasticity. To this end, we design an online learning method that can adaptively optimize the tradeoff without knowing the setting as a priori. Specifically, we first introduce the key hyperparameters that influence the tradeoff, e.g., knowledge distillation (KD) loss weights, learning rates, and classifier types. Then, we formulate the hyperparameter optimization process as an online Markov Decision Process (MDP) problem and propose a specific algorithm to solve it. We apply local estimated rewards and a classic bandit algorithm Exp3 to address the issues when applying online MDP methods to the CIL protocol. Our method consistently improves top-performing CIL methods in both TFH and TFS settings, e.g., boosting the average accuracy of TFH and TFS by 2.2 percentage points on ImageNet-Full, compared to the state-of-the-art. Code is provided at https://class-il.mpi-inf.mpg.de/online/


  • Learning to teach and learn for semi-supervised few-shot image classification

    by Xinzhe Li, Jianqiang Huang, Yaoyao Liu, Qin Zhou, Shibao Zheng, Bernt Schiele, Qianru Sun