mlan: language-based instruction tuning improves zero-shot generalization of multimodal large language models
Authors:
Preprint
Abstract
We present a novel instruction tuning recipe to improve the zero-shot task generalization of multimodal large language models. In contrast to existing instruction tuning mechanisms that heavily rely on visual instructions, our approach focuses on language-based instruction tuning, offering a distinct and more training efficient path for multimodal instruction tuning. We evaluate the performance of the proposed approach on 9 unseen datasets across both language and vision modalities. Our results show that our language-only instruction tuning is able to significantly improve the performance of two pretrained multimodal models based on Llama 2 and Vicuna on those unseen datasets. Interestingly, the language instruction following ability also helps unlock the models to follow vision instructions without explicit training. Compared to the state of the art multimodal instruction tuning approaches that are mainly based on visual instructions, our language-based method not only achieves superior performance but also significantly enhances training efficiency. For instance, the language-only instruction tuning produces competitive average performance across the evaluated datasets (with even better performance on language datasets) with significant training efficiency improvements (on average 4x), thanks to the striking reduction in the need for vision data. With a small number of visual instructions, this emerging language instruction following ability transfers well to the unseen vision datasets, outperforming the state of the art with greater training efficiency.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Instruction Tuning for Large Language Models: A SurveyACM Computing Surveys, 2026
- Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks2024
- Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks2024
- Visual Instruction Tuning with Polite FlamingoProceedings of the AAAI Conference on Artificial Intelligence, 2024
- MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning2023
- A-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge2022
- Did Aristotle Use a Laptop?A Question Answering Benchmark with Implicit Reasoning StrategiesTransactions of the Association for Computational Linguistics, 2021
- AI2D-RST: a multimodal corpus of 1000 primary school science diagramsLanguage Resources and Evaluation, 2020
- GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering2019
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering2017