SGL: Symbolic Goal Learning in a Hybrid, Modular Framework for Human Instruction Following
- 12 July 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 7 (4) , 10375-10382
Abstract
This paper investigates human instruction following for robotic manipulation via a hybrid, modular system with symbolic and connectionist elements. Symbolic methods build modular systems with semantic parsing and task planning modules for producing sequences of actions from natural language requests. Modern connectionist methods employ deep neural networks that learn visual and linguistic features for mapping inputs to a sequence of low-level actions, in an end-to-end fashion. The hybrid, modular system blends these two approaches to create a modular framework: it formulates instruction following as symbolic goal learning via deep neural networks followed by task planning via symbolic planners. Connectionist and symbolic modules are bridged with Planning Domain Definition Language. The vision-and-language learning network predicts its goal representation, which is sent to a planner for producing a task-completing action sequence. For improving the flexibility of natural language, we further incorporate implicit human intents with explicit human instructions. To learn generic features for vision and language, we propose to separately pretrain vision and language encoders on scene graph parsing and semantic textual similarity tasks. Benchmarking evaluates the impacts of different components of, or options for, the vision-and-language learning model and shows the effectiveness of pretraining strategies. Manipulation experiments conducted in the simulator AI2THOR show the robustness of the framework to novel scenarios.Keywords
All Related Versions
Funding Information
- NSF (2026611)
This publication has 36 references indexed in Scilit:
- INGRESS: Interactive visual grounding of referring expressionsThe International Journal of Robotics Research, 2020
- Your instruction may be crisp, but not clear to me!2019
- Toward Affordance Detection and Ranking on Novel Objects for Real-World Robotic ManipulationIEEE Robotics and Automation Letters, 2019
- Deep Modular Co-Attention Networks for Visual Question Answering2019
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- Neural Motifs: Scene Graph Parsing with Global Context2018
- Deep Residual Learning for Image Recognition2016
- Tell me Dave: Context-sensitive grounding of natural language to manipulation instructionsThe International Journal of Robotics Research, 2015
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- KnowRob: A knowledge processing infrastructure for cognition-enabled robotsThe International Journal of Robotics Research, 2013