PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 16262-16272
Abstract
With recent developments in Embodied Artificial Intel-ligence (EAI) research, there has been a growing demand for high-quality, large-scale interactive scene generation. While prior methods in scene synthesis have prioritized the naturalness and realism of the generated scenes, the physical plausibility and interactivity of scenes have been largely left unexplored. To address this disparity, we introduce PhyScene, a novel method dedicated to gener-ating interactive 3D scenes characterized by realistic lay-outs, articulated objects, and rich physical interactivity tai-lored for embodied agents. Based on a conditional diffusion model for capturing scene layouts, we devise novel physics-and interactivity-based guidance mechanisms that integrate constraints from object collision, room layout, and object reachability. Through extensive experiments, we demon-strate that PhyScene effectively leverages these guidance functions for physically interactable scene synthesis, out-performing existing state-of-the-art scene synthesis methods by a large margin. Our findings suggest that the scenes generated by PhyScene hold considerable potential for facilitating diverse skill acquisition among agents within in-teractive environments, thereby catalyzing further advance-ments in embodied AI research.Keywords
All Related Versions
This publication has 38 references indexed in Scilit:
- SceneFormer: Indoor Scene Generation with Transformers2021
- 3D-FUTURE: 3D Furniture Shape with TextUREInternational Journal of Computer Vision, 2021
- SG-VAE: Scene Grammar Variational Autoencoder to Generate New Indoor Scenes2020
- Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments2020
- RoboTHOR: An Open Simulation-to-Real Embodied AI Platform2020
- IoU Loss for 2D/3D Object Detection2019
- Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic GrammarsInternational Journal of Computer Vision, 2018
- Human-Centric Indoor Scene Synthesis Using Stochastic Grammar2018
- Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments2018
- ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes2017