Visual Instruction Tuning with Polite Flamingo
Authors:
Open Access
- 24 March 2024
- journal article
- research article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 38 (16) , 17745-17753
Abstract
Recent research has demonstrated that the multi-task fine-tuning of multi-modal Large Language Models (LLMs) using an assortment of annotated downstream vision-language datasets significantly enhances their performance. Yet, during this process, a side effect, which we termed as the "multi-modal alignment tax", surfaces. This side effect negatively impacts the model's ability to format responses appropriately - for instance, its "politeness" - due to the overly succinct and unformatted nature of raw annotations, resulting in reduced human preference. In this paper, we introduce Polite Flamingo, a multi-modal response rewriter that transforms raw annotations into a more appealing, "polite" format. Polite Flamingo is trained to reconstruct high-quality responses from their automatically distorted counterparts and is subsequently applied to a vast array of vision-language datasets for response rewriting. After rigorous filtering, we generate the PF-1M dataset and further validate its value by fine-tuning a multi-modal LLM with it. Combined with novel methodologies including U-shaped multi-stage tuning and multi-turn augmentation, the resulting model, Clever Flamingo, demonstrates its advantages in both multi-modal understanding and response politeness according to automated and human evaluations. Code and dataset are available at https://github.com/ChenDelong1999/polite-flamingoKeywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Otter: A Multi-Modal Model With In-Context Instruction TuningIEEE Transactions on Pattern Analysis and Machine Intelligence, 2025
- OCRBench: on the hidden mystery of OCR in large multimodal modelsScience China Information Sciences, 2024
- LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language ModelsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
- RemoteCLIP: A Vision Language Foundation Model for Remote SensingIEEE Transactions on Geoscience and Remote Sensing, 2024
- Visual Spatial ReasoningTransactions of the Association for Computational Linguistics, 2023
- Enhancing Chat Language Models by Scaling High-quality Instructional Conversations2023
- Self-Instruct: Aligning Language Models with Self-Generated Instructions2023
- A-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge2022