A survey on knowledge-enhanced multimodal learning
Open Access
- 9 September 2024
- journal article
- research article
- Published by Springer Nature in Artificial Intelligence Review
- Vol. 57 (10) , 1-80
Abstract
Multimodal learning has been a field of increasing interest, aiming to combine various modalities in a single joint representation. Especially in the area of visiolinguistic (VL) learning multiple models and techniques have been developed, targeting a variety of tasks that involve images and text. VL models have reached unprecedented performances by extending the idea of Transformers, so that both modalities can learn from each other. Massive pre-training procedures enable VL models to acquire a certain level of real-world understanding, although many gaps can be identified: the limited comprehension of commonsense, factual, temporal and other everyday knowledge aspects questions the extendability of VL tasks. Knowledge graphs and other knowledge sources can fill those gaps by explicitly providing missing information, unlocking novel capabilities of VL models. At the same time, knowledge graphs enhance explainability, fairness and validity of decision making, issues of outermost importance for such complex implementations. The current survey aims to unify the fields of VL representation learning and knowledge graphs, and provides a taxonomy and analysis of knowledge-enhanced VL models.Keywords
All Related Versions
Funding Information
- Hellenic Foundation for Research and Innovation (5537 (3rd Call for HFRI PhD Fellowships))
This publication has 381 references indexed in Scilit:
- Image captioning for effective use of language models in knowledge-based visual question answeringExpert Systems with Applications, 2022
- Kandinsky PatternsArtificial Intelligence, 2021
- Zero-Shot Visual Question Answering Using Knowledge Graph2021
- Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models2020
- YAGO 4: A Reason-able Knowledge Base2020
- Modeling Relational Data with Graph Convolutional Networks2018
- Modeling Context in Referring Expressions2016
- SPICE: Semantic Propositional Image Caption Evaluation2016
- Microsoft COCO: Common Objects in Context2014
- DBpedia: A Nucleus for a Web of Open Data2007