Ziya-Visual: Bilingual Large Vision-Language Model via Multi-Task Instruction Tuning
Preprint
Abstract
Recent advancements enlarge the capabilities of large language models (LLMs) in zero-shot image-to-text generation and understanding by integrating multi-modal inputs. However, such success is typically limited to English scenarios due to the lack of large-scale and high-quality non-English multi-modal resources, making it extremely difficult to establish competitive counterparts in other languages. In this paper, we introduce the Ziya-Visual series, a set of bilingual large-scale vision-language models (LVLMs) designed to incorporate visual semantics into LLM for multi-modal dialogue. Composed of Ziya-Visual-Base and Ziya-Visual-Chat, our models adopt the Querying Transformer from BLIP-2, further exploring the assistance of optimization schemes such as instruction tuning, multi-stage training and low-rank adaptation module for visual-language alignment. In addition, we stimulate the understanding ability of GPT-4 in multi-modal scenarios, translating our gathered English image-text datasets into Chinese and generating instruction-response through the in-context learning method. The experiment results demonstrate that compared to the existing LVLMs, Ziya-Visual achieves competitive performance across a wide range of English-only tasks including zero-shot image-text retrieval, image captioning, and visual question answering. The evaluation leaderboard accessed by GPT-4 also indicates that our models possess satisfactory image-text understanding and generation capabilities in Chinese multi-modal scenario dialogues. Code, demo and models are available at ~\url{https://huggingface.co/IDEA-CCNL/Ziya-BLIP2-14B-Visual-v1}.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Otter: A Multi-Modal Model With In-Context Instruction TuningIEEE Transactions on Pattern Analysis and Machine Intelligence, 2025
- Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection2024
- Recognize Anything: A Strong Image Tagging Model2024
- Improved Baselines with Visual Instruction Tuning2024
- MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning2023
- Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts2021
- Large-Scale Datasets for Going Deeper in Image Understanding2019
- COCO-CN for Cross-Lingual Image Tagging, Captioning, and RetrievalIEEE Transactions on Multimedia, 2019
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptionsTransactions of the Association for Computational Linguistics, 2014