Explaining Vision and Language through Graphs of Events in Space and Time
Open Access
- 2 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2818-2823
Abstract
Artificial Intelligence makes great advances today and starts to bridge the gap between vision and language. However, we are still far from understanding, explaining and controlling explicitly the visual content from a linguistic perspective, because we still lack a common explainable representation between the two domains. In this work we come to address this limitation and propose the Graph of Events in Space and Time (GEST), by which we can represent, create and explain, both visual and linguistic stories. We provide a theoretical justification of our model and an experimental validation, which proves that GEST can bring a solid complementary value along powerful deep learning models. In particular, GEST can help improve at the content-level the generation of videos from text, by being easily incorporated into our novel video generation engine. Additionally, by using efficient graph matching techniques, the GEST graphs can also improve the comparisons between texts at the semantic level.Keywords
All Related Versions
Funding Information
This publication has 23 references indexed in Scilit:
- Temporal Dynamic Graph LSTM for Action-Driven Video Object Detection2017
- Video Captioning With Attention-Based LSTM and Semantic ConsistencyIEEE Transactions on Multimedia, 2017
- Graph formulation of video activities for abnormal activity recognitionPattern Recognition, 2017
- SPICE: Semantic Propositional Image Caption Evaluation2016
- VQA: Visual Question Answering2015
- CIDEr: Consensus-based image description evaluation2015
- Glove: Global Vectors for Word Representation2014
- BLEU2001
- A dependency-based method for evaluating broad-coverage parsersNatural Language Engineering, 1998
- Rhetorical Structure Theory: Toward a functional theory of text organizationText & Talk, 1988