Yaoyao Liu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Yaoyao Liu has a total of 82 co-authors. Here are the top co-authors by number of shared publications:
- Wei Cao
University of Illinois Urbana-Champaign, USA
4 shared publications - Yingying Li
University of Illinois Urbana-Champaign
3 shared publications Qianru Sun
2 shared publications- Yulun Wu
University of Illinois Urbana-Champaign
2 shared publications - Adam Kortylewski
Max Planck Institute for Informatics
1 shared publication - Alan Yuille
Johns Hopkins University
1 shared publication
Latest Publications
- Open Access
- 19 October 2026
- Published by MDPI AG in Preprints.org
Native Real-Time Interactive Omni Intelligence: A Unified Framework
The rapid progress of large language models has established strong foundations for reasoning, knowledge, and generation, while the integration of vision, audio, video, and other modalities has extended these capabilities toward increasingly unified intelligence across modalities, which we refer to as Omni Intelligence. Yet, real-world assistants must do more than process completed multimodal prompts. They must perceive evolving streams, respond while new evidence is arriving, handle hesitation, overlap, interruption, and changing intent, and coordinate reasoning or tool use that may proceed asynchronously. We refer to this broader capability as \( \textbf{Interactive Omni Intelligence} \): the ability to maintain and exercise omni intelligence under the temporal and adaptive demands of real-time interaction. The question guiding this paper is: How can foundation-model capabilities be carried into real-time interaction while preserving both intelligence and interactivity?We develop a conceptual framework connecting two development paths. One adapts pretrained language or omni models through streaming interfaces, modality modules, post-training, and orchestration. The other develops model-native systems that learn temporal structure, continuous multimodal processing, generation, and interaction control. Across both paths, we compare representations, architectures, fusion, data, training objectives, serving, and evaluation. Their comparison reveals an Intelligence--Interactivity Trade-off between preserving reasoning and knowledge, and meeting the timing, coordination, and resource demands of continuous operation. We examine Multi-Rate Interactive Intelligence as a cross-cutting direction, including foreground--background designs that coordinate fast interaction with slower reasoning, retrieval, memory, and tools. Finally, we identify open problems in evolving state, persistent and proactive interaction, joint evaluation, and safe open-world operation. The resulting framework connects capability preservation, interaction control, and multi-rate coordination.- 19 October 2026
- Published by MDPI AG in Preprints.org
Open AccessNative Real-Time Interactive Omni Intelligence: A Unified Framework
The rapid progress of large language models has established strong foundations for reasoning, knowledge, and generation, while the integration of vision, audio, video, and other modalities has extended these capabilities toward increasingly unified intelligence across modalities, which we refer to as Omni Intelligence. Yet, real-world assistants must do more than process completed multimodal prompts. They must perceive evolving streams, respond while new evidence is arriving, handle hesitation, overlap, interruption, and changing intent, and coordinate reasoning or tool use that may proceed asynchronously. We refer to this broader capability as \( \textbf{Interactive Omni Intelligence} \): the ability to maintain and exercise omni intelligence under the temporal and adaptive demands of real-time interaction. The question guiding this paper is: How can foundation-model capabilities be carried into real-time interaction while preserving both intelligence and interactivity?We develop a conceptual framework connecting two development paths. One adapts pretrained language or omni models through streaming interfaces, modality modules, post-training, and orchestration. The other develops model-native systems that learn temporal structure, continuous multimodal processing, generation, and interaction control. Across both paths, we compare representations, architectures, fusion, data, training objectives, serving, and evaluation. Their comparison reveals an Intelligence--Interactivity Trade-off between preserving reasoning and knowledge, and meeting the timing, coordination, and resource demands of continuous operation. We examine Multi-Rate Interactive Intelligence as a cross-cutting direction, including foreground--background designs that coordinate fast interaction with slower reasoning, retrieval, memory, and tools. Finally, we identify open problems in evolving state, persistent and proactive interaction, joint evaluation, and safe open-world operation. The resulting framework connects capability preservation, interaction control, and multi-rate coordination. - 15 September 2026
- Published by Springer Nature
- p. 405-424
Active View Selection with Perturbed Gaussian Ensemble for Tomographic Reconstruction
- 15 September 2026
- Published by Springer Nature
- p. 405-424
Active View Selection with Perturbed Gaussian Ensemble for Tomographic Reconstruction
- 13 September 2026
- Published by Springer Nature
- p. 422-440
How To Teach Large Multimodal Models New Skills?
- 13 September 2026
- Published by Springer Nature
- p. 422-440
How To Teach Large Multimodal Models New Skills?
- 12 September 2026
- Published by Springer Nature
- p. 510-525
AC3S: Adaptive Conditioning for 3D-Aware Synthetic Data Generation
- 12 September 2026
- Published by Springer Nature
- p. 510-525
AC3S: Adaptive Conditioning for 3D-Aware Synthetic Data Generation
- Open Access
- 1 August 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (8) , 9519-9536
Motion2VecSets: Non-Rigid Shape Reconstruction and Tracking With 4D Latent Set Diffusion
We introduce Motion2VecSets, a 4D diffusion model for dynamic surface mesh generation from various ambiguous observations, including a sequence of RGB images, sparse and partial point clouds, and low-resolution voxel grids. While recent methods using neural field representations have shown success in modeling non-rigid objects, conventional feed-forward architectures struggle with noisy, partial, or sparse observations due to their deterministic nature. To address the inherent one-to-many mapping problem, we introduce a diffusion model that explicitly learns the shape and motion distribution of non-rigid objects through an iterative denoising process of compressed latent representations. The diffusion-based priors provide more plausible and diverse reconstructions under ambiguous conditions. Instead of relying on global latent codes, we represent 4D dynamics using latent sets. This novel 4D representation captures local shape and deformation patterns, leading to more accurate non-linear motion capture and significantly improving generalization capacity to unseen motions and identities. For temporally coherent tracking, we jointly denoise latent sets across frames and enable cross-frame information exchange. To reduce computational cost, we design an interleaved spatial-temporal attention block that alternately aggregates deformation latents along spatial and temporal dimensions. Extensive experiments on datasets of humans, animals, and articulated objects demonstrate that Motion2VecSets outperforms prior methods in reconstructing and tracking non-rigid deformations from various imperfect observations. Our implementation is available at https://vveicao.github.io/projects/Motion2VecSets/.
- 1 August 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (8) , 9519-9536
Open AccessMotion2VecSets: Non-Rigid Shape Reconstruction and Tracking With 4D Latent Set Diffusion
We introduce Motion2VecSets, a 4D diffusion model for dynamic surface mesh generation from various ambiguous observations, including a sequence of RGB images, sparse and partial point clouds, and low-resolution voxel grids. While recent methods using neural field representations have shown success in modeling non-rigid objects, conventional feed-forward architectures struggle with noisy, partial, or sparse observations due to their deterministic nature. To address the inherent one-to-many mapping problem, we introduce a diffusion model that explicitly learns the shape and motion distribution of non-rigid objects through an iterative denoising process of compressed latent representations. The diffusion-based priors provide more plausible and diverse reconstructions under ambiguous conditions. Instead of relying on global latent codes, we represent 4D dynamics using latent sets. This novel 4D representation captures local shape and deformation patterns, leading to more accurate non-linear motion capture and significantly improving generalization capacity to unseen motions and identities. For temporally coherent tracking, we jointly denoise latent sets across frames and enable cross-frame information exchange. To reduce computational cost, we design an interleaved spatial-temporal attention block that alternately aggregates deformation latents along spatial and temporal dimensions. Extensive experiments on datasets of humans, animals, and articulated objects demonstrate that Motion2VecSets outperforms prior methods in reconstructing and tracking non-rigid deformations from various imperfect observations. Our implementation is available at https://vveicao.github.io/projects/Motion2VecSets/.
- Top Cited PapersOpen Access
- 19 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-12
FreeOrbit4D: Training-Free Arbitrary Camera Redirection for Monocular Videos via Foreground-Complete 4D Reconstruction
Camera redirection aims to replay a dynamic scene from a single monocular video under a user-specified camera trajectory. However, large-angle redirection is inherently ill-posed: a monocular video captures only a narrow spatio-temporal view of a dynamic 3D scene, providing severely limited observations of the underlying 4D world. The key challenge is therefore to recover a complete and coherent plenoptic representation from this limited input, with consistent geometry and coherent motion. While recent diffusion-based methods achieve impressive visual generation quality, they often break down under large-angle viewpoint changes far from the original trajectory, where missing visual grounding leads to severe geometric ambiguity and temporal inconsistency. To address this, we present FreeOrbit4D, an effective training-free framework that tackles this geometric ambiguity by recovering a foreground-complete 4D proxy as structural grounding for video generation. We obtain this proxy by decoupling foreground and background reconstructions: we unproject the monocular video into a static background and partial foreground point clouds in a unified global space, then leverage an object-centric multi-view diffusion model to synthesize multi-view images and reconstruct complete foreground point clouds in canonical object space. By aligning the canonical foreground point cloud to the global scene space via dense pixel-synchronized 3D–3D correspondences and projecting the foreground-complete 4D proxy onto target camera viewpoints, we provide geometric scaffolds (e.g., depth/visibility cues) that guide a conditional video diffusion model. Extensive experiments show that FreeOrbit4D produces more faithful and temporally coherent redirected videos under challenging large-angle trajectories, and our foreground-complete 4D proxy further opens a potential avenue for practical applications such as edit propagation and 4D data generation. Project page: https://freeorbit4d.vision.ischool.illinois.edu/
- 19 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-12
Top Cited PapersOpen AccessFreeOrbit4D: Training-Free Arbitrary Camera Redirection for Monocular Videos via Foreground-Complete 4D Reconstruction
Camera redirection aims to replay a dynamic scene from a single monocular video under a user-specified camera trajectory. However, large-angle redirection is inherently ill-posed: a monocular video captures only a narrow spatio-temporal view of a dynamic 3D scene, providing severely limited observations of the underlying 4D world. The key challenge is therefore to recover a complete and coherent plenoptic representation from this limited input, with consistent geometry and coherent motion. While recent diffusion-based methods achieve impressive visual generation quality, they often break down under large-angle viewpoint changes far from the original trajectory, where missing visual grounding leads to severe geometric ambiguity and temporal inconsistency. To address this, we present FreeOrbit4D, an effective training-free framework that tackles this geometric ambiguity by recovering a foreground-complete 4D proxy as structural grounding for video generation. We obtain this proxy by decoupling foreground and background reconstructions: we unproject the monocular video into a static background and partial foreground point clouds in a unified global space, then leverage an object-centric multi-view diffusion model to synthesize multi-view images and reconstruct complete foreground point clouds in canonical object space. By aligning the canonical foreground point cloud to the global scene space via dense pixel-synchronized 3D–3D correspondences and projecting the foreground-complete 4D proxy onto target camera viewpoints, we provide geometric scaffolds (e.g., depth/visibility cues) that guide a conditional video diffusion model. Extensive experiments show that FreeOrbit4D produces more faithful and temporally coherent redirected videos under challenging large-angle trajectories, and our foreground-complete 4D proxy further opens a potential avenue for practical applications such as edit propagation and 4D data generation. Project page: https://freeorbit4d.vision.ischool.illinois.edu/
- 11 September 2025
- Published by Elsevier in Neurocomputing
- Vol. 656
A rate-dependent coreset selector for continual learning on time-varying data distributions
- 11 September 2025
- Published by Elsevier in Neurocomputing
- Vol. 656
A rate-dependent coreset selector for continual learning on time-varying data distributions
- 29 October 2024
- Published by Springer Nature
- p. 357-374
iNeMo: Incremental Neural Mesh Models for Robust Class-Incremental Learning
- 29 October 2024
- Published by Springer Nature
- p. 357-374
iNeMo: Incremental Neural Mesh Models for Robust Class-Incremental Learning
- Open Access
- 26 June 2023
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 37 (7) , 8906-8913
Online Hyperparameter Optimization for Class-Incremental Learning
Class-incremental learning (CIL) aims to train a classification model while the number of classes increases phase-by-phase. An inherent challenge of CIL is the stability-plasticity tradeoff, i.e., CIL models should keep stable to retain old knowledge and keep plastic to absorb new knowledge. However, none of the existing CIL models can achieve the optimal tradeoff in different data-receiving settings—where typically the training-from-half (TFH) setting needs more stability, but the training-from-scratch (TFS) needs more plasticity. To this end, we design an online learning method that can adaptively optimize the tradeoff without knowing the setting as a priori. Specifically, we first introduce the key hyperparameters that influence the tradeoff, e.g., knowledge distillation (KD) loss weights, learning rates, and classifier types. Then, we formulate the hyperparameter optimization process as an online Markov Decision Process (MDP) problem and propose a specific algorithm to solve it. We apply local estimated rewards and a classic bandit algorithm Exp3 to address the issues when applying online MDP methods to the CIL protocol. Our method consistently improves top-performing CIL methods in both TFH and TFS settings, e.g., boosting the average accuracy of TFH and TFS by 2.2 percentage points on ImageNet-Full, compared to the state-of-the-art. Code is provided at https://class-il.mpi-inf.mpg.de/online/
- 26 June 2023
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 37 (7) , 8906-8913
Open AccessOnline Hyperparameter Optimization for Class-Incremental Learning
Class-incremental learning (CIL) aims to train a classification model while the number of classes increases phase-by-phase. An inherent challenge of CIL is the stability-plasticity tradeoff, i.e., CIL models should keep stable to retain old knowledge and keep plastic to absorb new knowledge. However, none of the existing CIL models can achieve the optimal tradeoff in different data-receiving settings—where typically the training-from-half (TFH) setting needs more stability, but the training-from-scratch (TFS) needs more plasticity. To this end, we design an online learning method that can adaptively optimize the tradeoff without knowing the setting as a priori. Specifically, we first introduce the key hyperparameters that influence the tradeoff, e.g., knowledge distillation (KD) loss weights, learning rates, and classifier types. Then, we formulate the hyperparameter optimization process as an online Markov Decision Process (MDP) problem and propose a specific algorithm to solve it. We apply local estimated rewards and a classic bandit algorithm Exp3 to address the issues when applying online MDP methods to the CIL protocol. Our method consistently improves top-performing CIL methods in both TFH and TFS settings, e.g., boosting the average accuracy of TFH and TFS by 2.2 percentage points on ImageNet-Full, compared to the state-of-the-art. Code is provided at https://class-il.mpi-inf.mpg.de/online/
- 1 November 2021
- Published by Elsevier in Computer Vision and Image Understanding
- Vol. 212
Learning to teach and learn for semi-supervised few-shot image classification
- 1 November 2021
- Published by Elsevier in Computer Vision and Image Understanding
- Vol. 212
Learning to teach and learn for semi-supervised few-shot image classification