From VEDs to VLMs: Assessing Fine-Tuning Efficiency and Metric Reliability in Brazilian Portuguese Image Captioning
Preprint
- 19 April 2026
- preprint
- Published by Elsevier in SSRN Electronic Journal
Abstract
The adaptability and robustness of Visual Language Models (VLMs) to low-resource linguistic contexts remain a critical frontier in multimodal machine learning. This work evaluates the performance of state-of-the-art architectures, including ViTucano, PaliGemma, Phi-3 V, and LLaMA 3.2 V, in the context of Brazilian Portuguese image captioning (IC). By contrasting native human-annotated captions with machine-translated equivalents from the Flickr30K dataset, we delineate the performance gain and drop of using a translated corpus for fine-tuning versus its version annotated by native speakers, along with an additional comparison to the zero-shot setting. Furthermore, we demonstrate that VLMs fine-tuned under restricted conditions significantly outperform Vision Encoder-Decoder (VED) models, with ViTucano 2B, Phi-3 V, and PaliGemma achieving the highest results across translated and native data. Finally, our metrics adequacy analysis suggests that CLIP-Score is more robust than traditional unimodal metrics in distinguishing correct captions from incorrect, randomly selected captions.Keywords
This publication has 8 references indexed in Scilit:
- A Unified Agentic Framework for Evaluating Conditional Image Generation2025
- Deep Learning Approaches on Image Captioning: A ReviewACM Computing Surveys, 2023
- QLoRA: Efficient Finetuning of Quantized LLMs2023
- Visuals to Text: A Comprehensive Review on Automatic Image CaptioningIEEE/CAA Journal of Automatica Sinica, 2022
- From Show to Tell: A Survey on Deep Learning-Based Image CaptioningIEEE Transactions on Pattern Analysis and Machine Intelligence, 2022
- #PraCegoVer: A Large Dataset for Image Captioning in PortugueseData, 2022
- CLIPScore: A Reference-free Evaluation Metric for Image Captioning2021
- A Comprehensive Survey of Deep Learning for Image CaptioningACM Computing Surveys, 2019