OneLLM: One Framework to Align All Modalities with Language
Authors:
Top Cited Papers
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 26574-26585
Abstract
Multimodal large language models (MLLMs) have gained significant attention due to their strong multimodal understanding capability. However, existing works rely heavily on modality-specific encoders, which usually differ in architecture and are limited to common modalities. In this paper, we present OneLLM, an MLLM that aligns eight modalities to language using a unified framework. We achieve this through a unified multimodal encoder and a progressive multimodal alignment pipeline. In detail, we first train an image projection module to connect a vision encoder with LLM. Then, we build a universal projection module (UPM) by mixing multiple image projection modules and dynamic routing. Finally, we progressively align more modalities to LLM with the UPM. To fully leverage the potential of OneLLM in following instructions, we also curated a comprehensive multimodal instruction dataset, including 2M items from image, audio, video, point cloud, depth/normal map, IMU and fMRI brain activity. OneLLM is evaluated on 25 diverse benchmarks, encompassing tasks such as multimodal captioning, question answering and reasoning, where it delivers excellent performance. Code, data, model and online demo are available at https://github.com/csuhan/OneLLM.Keywords
All Related Versions
This publication has 42 references indexed in Scilit:
- A-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge2022
- Video Dialog as Conversation About Objects Living in Space-Time2022
- Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Models2022
- CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioningNeurocomputing, 2022
- A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligenceNature Neuroscience, 2021
- VizWiz Grand Challenge: Answering Visual Questions from Blind People2018
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering2017
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language2016
- SUN RGB-D: A RGB-D scene understanding benchmark suite2015