Visual Instruction Tuning with Polite Flamingo
Authors:
Preprint
Abstract
Recent research has demonstrated that the multi-task fine-tuning of multi-modal Large Language Models (LLMs) using an assortment of annotated downstream vision-language datasets significantly enhances their performance. Yet, during this process, a side effect, which we termed as the "multi-modal alignment tax", surfaces. This side effect negatively impacts the model's ability to format responses appropriately -- for instance, its "politeness" -- due to the overly succinct and unformatted nature of raw annotations, resulting in reduced human preference. In this paper, we introduce Polite Flamingo, a multi-modal response rewriter that transforms raw annotations into a more appealing, "polite" format. Polite Flamingo is trained to reconstruct high-quality responses from their automatically distorted counterparts and is subsequently applied to a vast array of vision-language datasets for response rewriting. After rigorous filtering, we generate the PF-1M dataset and further validate its value by fine-tuning a multi-modal LLM with it. Combined with novel methodologies including U-shaped multi-stage tuning and multi-turn augmentation, the resulting model, Clever Flamingo, demonstrates its advantages in both multi-modal understanding and response politeness according to automated and human evaluations.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- OCRBench: on the hidden mystery of OCR in large multimodal modelsScience China Information Sciences, 2024
- LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language ModelsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
- On Scaling Up a Multilingual Vision and Language Model2024
- Improved Baselines with Visual Instruction Tuning2024
- RemoteCLIP: A Vision Language Foundation Model for Remote SensingIEEE Transactions on Geoscience and Remote Sensing, 2024
- What does CLIP know about a red circle? Visual prompt engineering for VLMs2023
- Information Recovery-Driven Deep Incomplete Multiview Clustering NetworkIEEE Transactions on Neural Networks and Learning Systems, 2023
- Localized Sparse Incomplete Multi-View ClusteringIEEE Transactions on Multimedia, 2022
- Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image RetrievalIEEE Transactions on Geoscience and Remote Sensing, 2021
- Exploring Models and Data for Remote Sensing Image Caption GenerationIEEE Transactions on Geoscience and Remote Sensing, 2017