Detailed Image Captioning and Hashtag Generation
Open Access
- 28 November 2024
- journal article
- research article
- Published by MDPI AG in Future Internet
- Vol. 16 (12) , 444
Abstract
This article presents CapFlow, an integrated approach to detailed image captioning and hashtag generation. Based on a thorough performance evaluation, the image captioning model utilizes a fine-tuned vision-language model with Low-Rank Adaptation (LoRA), while the hashtag generation employs the keyword extraction method. We evaluated the state-of-the-art image captioning models using both traditional metrics (BLEU, METEOR, ROUGE-L, and CIDEr) and the specialized CAPTURE metric for detailed captions. The hashtag generation models were assessed using precision, recall, and F1-score. The proposed method demonstrates competitive results against larger models while maintaining efficiency suitable for real-time applications. The image captioning model outperforms the base Florence-2 model and favorably compares with larger models. The KeyBERT implementation for hashtag generation surpasses other keyword extraction methods in both accuracy and speed. This work contributes to the field of AI-assisted content analysis and generation, offering insights into the practical implementation of advanced vision-language models for detailed image understanding and relevant tag generation.Keywords
This publication has 40 references indexed in Scilit:
- A Comparative Assessment of Unsupervised Keyword Extraction ToolsIEEE Access, 2023
- Exploiting Multiple Sequence Lengths in Fast End to End Training for Image Captioning2023
- Deep Learning Approaches on Image Captioning: A ReviewACM Computing Surveys, 2023
- Image Captions and Hashtags Generation Using Deep Learning Approach2022
- DaViT: Dual Attention Vision Transformers2022
- GRIT: Faster and Better Image Captioning Transformer Using Dual Visual Features2022
- XGPT: Cross-modal Generative Pre-Training for Image Captioning2021
- Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks2020
- YAKE! Keyword extraction from single documents using multiple local featuresInformation Sciences, 2020
- SPICE: Semantic Propositional Image Caption Evaluation2016