Generating Executable Action Plans with Environmentally-Aware Language Models
Authors:
- 1 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3568-3575
Abstract
Large Language Models (LLMs) trained using massive text datasets have recently shown promise in generating action plans for robotic agents from high-level text queries. However, these models typically do not consider the robot's environment, resulting in generated plans that may not actually be executable, due to ambiguities in the planned actions or environmental constraints. In this paper, we propose an approach to generate environmentally-aware action plans that agents are better able to execute. Our approach involves integrating environmental objects and object relations as additional inputs into LLM action plan generation to provide the system with an awareness of its surroundings, resulting in plans where each generated action is mapped to objects present in the scene. We also design a novel scoring function that, along with generating the action steps and associating them with objects, helps the system disambiguate among object instances and take into account their states. We evaluated our approach using the VirtualHome simulator and the ActivityPrograms knowledge base and found that action plans generated from our system had a 310% improvement in executability and a 147% improvement in correctness over prior work. The complete code and a demo of our method is publicly available at https://github.com/hri-ironlab/scene_aware_language_planner.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Differentiable Physics and Stable Modes for Tool-Use and Manipulation Planning - Extended Abtract2019
- Language Models as Knowledge Bases?2019
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks2019
- VirtualHome: Simulating Household Activities Via Programs2018
- Visual Semantic Planning Using Deep Successor Representations2017
- Tell me Dave: Context-sensitive grounding of natural language to manipulation instructionsThe International Journal of Robotics Research, 2015
- Environment-Driven Lexicon Induction for High-Level Instructions2015
- Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to ActionsTransactions of the Association for Computational Linguistics, 2013
- Understanding and executing instructions for everyday manipulation tasks from the World Wide Web2010
- Strips: A new approach to the application of theorem proving to problem solvingArtificial Intelligence, 1971