tagE: Enabling an Embodied Agent to Understand Human Instructions
Preprint
Abstract
Natural language serves as the primary mode of communication when an intelligent agent with a physical presence engages with human beings. While a plethora of research focuses on natural language understanding (NLU), encompassing endeavors such as sentiment analysis, intent prediction, question answering, and summarization, the scope of NLU directed at situations necessitating tangible actions by an embodied agent remains limited. The inherent ambiguity and incompleteness inherent in natural language present challenges for intelligent agents striving to decipher human intention. To tackle this predicament head-on, we introduce a novel system known as task and argument grounding for Embodied agents (tagE). At its core, our system employs an inventive neural network model designed to extract a series of tasks from complex task instructions expressed in natural language. Our proposed model adopts an encoder-decoder framework enriched with nested decoding to effectively extract tasks and their corresponding arguments from these intricate instructions. These extracted tasks are then mapped (or grounded) to the robot's established collection of skills, while the arguments find grounding in objects present within the environment. To facilitate the training and evaluation of our system, we have curated a dataset featuring complex instructions. The results of our experiments underscore the prowess of our approach, as it outperforms robust baseline models.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Talk-to-Resolve: Combining scene understanding and spatial dialogue to resolve granular task ambiguity for a collocated robotRobotics and Autonomous Systems, 2022
- Causality extraction based on self-attentive BiLSTM-CRF with transferred embeddingsNeurocomputing, 2021
- Effective Modeling of Encoder-Decoder Architecture for Joint Entity and Relation ExtractionProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Position-Aware Tagging for Aspect Sentiment Triplet Extraction2020
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- Grounded language interpretation of robotic commands through structured learningArtificial Intelligence, 2020
- Your instruction may be crisp, but not clear to me!2019
- Situated Human–Robot Collaboration: predicting intent from grounded natural language2018
- A Question-Focused Multi-Factor Attention Network for Question AnsweringProceedings of the AAAI Conference on Artificial Intelligence, 2018
- Jointly Extracting Event Triggers and Arguments by Dependency-Bridge RNN and Tensor-Based Argument InteractionProceedings of the AAAI Conference on Artificial Intelligence, 2018