Multimodal Foundation Models for Unified Image, Video and Text Understanding
Authors:
- Ayodele R. AkinyeleAAAyodele R. Akinyele
- Oseghale IhayereOIOseghale Ihayere
- Osayi EromhonseleOEOsayi Eromhonsele
- Ehisuoria AigbogunEAEhisuoria Aigbogun
- Adebayo Nurudeen KalejaiyeAKAdebayo Nurudeen Kalejaiye
- Oluwole Olakunle AjayiOAOluwole Olakunle Ajayi
Open Access
- 30 November 2024
- journal article
- Published by Open Access Research Journals Publication in Open Access Research Journal of Science and Technology
- Vol. 12 (2) , 081-093
Abstract
Advanced models can now interpret and understand photos, videos, and text. Conventional AI models focused on image classification, text analysis, and video processing. Multimodal foundation models combine data analysis in one framework to meet the requirement for more integrated AI systems. The models may learn joint representations from several modalities to generate text from images, analyze movies with textual context, and answer visual queries. Cross-modal learning's theoretical foundations and architectural advances have helped multimodal foundation models grow rapidly. This study explores their success. Transformer-based architectures have profoundly changed AI model data modalities. Self-attention and contrastive learning help the models align and integrate data across modalities, improving data understanding. The study analyses well-known multimodal models CLIP, ALIGN, Flamingo, and Video BERT, emphasizing on their design, training, and performance across tasks. Their performance in caption generation, video-text retrieval, and visual reasoning has led to more adaptable AI systems that can handle complex real-world scenarios. Despite promising results, multimodal learning faces various obstacles. To create effective models, you need substantial, high-quality datasets. Computers struggle to handle many data formats simultaneously, and bias and interpretability difficulties arise. The limitations and ethical implications of multimodal models in healthcare and autonomous systems are examined in this research. This study investigates the future of multimodal foundation models, focused on reducing processing, enhancing model fairness, and applying them to audio, sensor data, and robotics. Understanding and integrating multimodal information is essential for creating more intuitive and intelligent systems, therefore unified multimodal models could change human-computer interaction. Overall, multimodal foundation models drive the search for generalized and adaptable AI systems. These models' capacity to combine picture, video, and text data could alter many applications, driving creativity across sectors and stimulating AI research.Keywords
This publication has 59 references indexed in Scilit:
- Multimodal Human–Robot Interaction for Human‐Centric Smart Manufacturing: A SurveyAdvanced Intelligent Systems, 2023
- Interpreting Black-Box Models: A Review on Explainable Artificial IntelligenceCognitive Computation, 2023
- Artificial intelligence and smart vision for building and construction 4.0: Machine and deep learning methods and applicationsAutomation in Construction, 2022
- A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasetsThe Visual Computer, 2021
- Communication Models in Human–Robot Interaction: An Asymmetric MODel of ALterity in Human–Robot Interaction (AMODAL-HRI)International Journal of Social Robotics, 2021
- Modeling dengue vector population with earth observation data and a generalized linear modelActa Tropica, 2021
- Pre-trained models: Past, present and futureAI Open, 2021
- Multimodal machine translation through visuals and speechMachine Translation, 2020
- Grassland ecosystem services: a systematic review of research advances and future directionsLandscape Ecology, 2020
- Visual question answering: A survey of methods and datasetsComputer Vision and Image Understanding, 2017