Fangzhou Qiu

Reconstructing geometry, appearance, and motion from videos is a central problem in computer vision and world modeling. In the monocular dynamic setting, this problem is severely ill-posed because the underlying 3D structure and time-varying motion must be inferred from sparse and self-motion views. Mitigating this ill-posedness requires a representation that is both temporally coherent and sufficiently constrained for monocular optimization. We introduce Gaussian Sequences, a framework that converts a monocular video into a temporally coherent sequence of 3D Gaussian primitives for dynamic reconstruction and 3D tracking. At the core of the framework is a global-local SE(3) dynamic field. Global motion patterns provide a compact representation of coherent scene-level dynamics, while local residual motion controllers add bounded corrections for fine non-rigid deformation. To further reduce monocular ambiguity, we leverage foundation-model priors as initialization cues and complementary constraints that guide geometry and motion recovery under weak observations. As an application, Gaussian sequences serve as spatio-temporal observations for differentiable particle simulation, enabling physical parameter identification from monocular videos and action-conditioned dynamics prediction. Experiments on real-world monocular videos show that our Gaussian sequences outperform strong dynamic reconstruction baselines in novel-view synthesis and point tracking. Downstream particle simulation of deformable objects further validates that the reconstructed sequences provide useful information for predictive physical modeling.

Perspective ambiguity widely exists in human-robot interaction, especially in the language-driven robot manipulation. It is easy to place the object at a wrong position just because of different perspectives. Thus, placement location determination is required in language-driven robot manipulation. In this letter, we introduced a Language-Driven Perspective-based Pick-and-Place Algorithm (LD3PA) to solve this problem. By taking natural language description, image and point cloud from Kinects, a perspective disambiguation network using multi-head self-attention is proposed to infer the user’s perspective. A command analysis network is also designed to decode object’s role and relation from user’s commands. A placement optimization algorithm based on convolution is proposed to find the final placement location and orientation of the picked objects considering the edge of the table and the objects already on the table. Experiments show that our proposed LD3PA could not only correctly disambiguate the user’s perspective, but also find a good location for object placement. The perspective disambiguation ability outperforms human in the comparable experiment. Finally, we implement the algorithm on physical Panda Robot. The result shows that our proposed LD3PA could fuse the multi-modal information to solve the perspective ambiguity and placement optimization problem in human-robot interaction.