Scene-Aware Activity Program Generation with Language Guidance
Open Access
- 5 December 2023
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 42 (6) , 1-16
Abstract
We address the problem of scene-aware activity program generation, which requires decomposing a given activity task into instructions that can be sequentially performed within a target scene to complete the activity. While existing methods have shown the ability to generate rational or executable programs, generating programs with both high rationality and executability still remains a challenge. Hence, we propose a novel method where the key idea is to explicitly combine the language rationality of a powerful language model with dynamic perception of the target scene where instructions are executed, to generate programs with high rationality and executability. Our method iteratively generates instructions for the activity program. Specifically, a two-branch feature encoder operates on a language-based and graph-based representation of the current generation progress to extract language features and scene graph features, respectively. These features are then used by a predictor to generate the next instruction in the program. Subsequently, another module performs the predicted action and updates the scene for perception in the next iteration. Extensive evaluations are conducted on the VirtualHome-Env dataset, showing the advantages of our method over previous work. Key algorithmic designs are validated through ablation studies, and results on other types of inputs are also presented to show the generalizability of our method.Keywords
Funding Information
- Tencent AI Lab Rhino-Bird Focused Research Program (RBFR2022013)
- NSFC (62322207, U21B2023, U2001206)
- GD Natural Science Foundation (2021B1515020085)
- Shenzhen Science and Technology Program (RCYX20210609103121030)
This publication has 21 references indexed in Scilit:
- Dynamic graph convolutional networksPattern Recognition, 2020
- Temporal Network Embedding with Micro- and Macro-dynamics2019
- Synthesizing Environment-Aware Activities via Activity Sketches2019
- VirtualHome: Simulating Household Activities Via Programs2018
- ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes2017
- Tell me Dave: Context-sensitive grounding of natural language to manipulation instructionsThe International Journal of Robotics Research, 2015
- Environment-Driven Lexicon Induction for High-Level Instructions2015
- Manipulation action tree bank: A knowledge resource for humanoids2014
- Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to ActionsTransactions of the Association for Computational Linguistics, 2013
- Everything robots always wanted to know about housework (but were afraid to ask)2012