Generating Explanations for Embodied Action Decision from Visual Observation
Open Access
- 26 October 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 2838-2846
Abstract
Getting trust is crucial for embodied agents (such as robots and autonomous vehicles) to collaborate with human beings, especially non-experts. The most direct way for mutual understanding is through natural language explanation. Existing researches consider generating visual explanations for object recognition, while the exploration of explaining embodied decisions remains vacant. In this paper, we study generating action decisions and explanations based on visual observation. Distinct to explanations for recognition, justifying an action needs to show why it's better than other actions. Besides, the understanding of scene structure is required since the agent needs to interact with the environment (e.g. navigation, moving objects). We introduce a new dataset THOR-EAE (Embodied Action Explanation) collected based on AI2-THOR simulator. The dataset consists of over 840,000 egocentric images of indoor embodied observation which are annotated with the optimal action labels and explanation sentences. An explainable decision-making criterion is developed considering scene layout and action attributes for efficient annotation. We propose a graph action justification model, exploiting graph neural networks for obstacle-surroundings relations representation and justifying the actions under the guidance of decision results. Experimental results on THOR-EAE dataset showcase its challenge and the effectiveness of the proposed method.Keywords
Funding Information
- National Key Research and Development Project of New Generation Artificial Intelligence of China (2018AAA0102504)
This publication has 25 references indexed in Scilit:
- Mask R-CNN2017
- Scene Recognition with CNNs: Objects, Scales and Dataset Bias2016
- Image Captioning with Deep Bidirectional LSTMs2016
- Generating Visual Explanations2016
- Learning Deep Representations of Fine-Grained Visual Descriptions2016
- Long-term recurrent convolutional networks for visual recognition and description2015
- Show and tell: A neural image caption generator2015
- CIDEr: Consensus-based image description evaluation2015
- Microsoft COCO: Common Objects in Context2014
- Automated Flower Classification over a Large Number of Classes2008