Video Captioning Method Based on Semantic Topic Association
Open Access
- 25 February 2025
- journal article
- research article
- Published by MDPI AG in Electronics
- Vol. 14 (5) , 905
Abstract
To address the issue in encoder–decoder-based video captioning method where semantic topics are overlooked in guiding the video captioning, leading to a semantic mismatch between video content and generated captions, a video captioning method based on semantic topic association is proposed. The core of this method lies in the designed semantic topic module, which is capable of capturing more detailed topic information on objects and actions in videos, and establishing a bridge between video representations and semantic topics, thereby selectively guiding the generation of video descriptions. During the decoding phase, an optimized NGPT kernel sampling decoding strategy is employed, constructing an NGPT model based on Top-P kernel sampling to jointly decode semantic information and generate captions. This model fully leverages the topic information in the video, narrowing the “semantic gap”, mitigating redundancy issues during decoding, and making the generated text descriptions that are more aligned with the topic content of the video. Ultimately, experiments were carried out using two publicly accessible datasets: MSVD and MSR-VTT. The findings reveal that our proposed approach surpasses several leading methods, achieving enhancements of 5.3% in BLEU4, 3.1% in METEOR, 2.4% in ROUGE_L, and 13.1% in CIDEr on the MSVD dataset, as well as improvements of 0.9%, 0.4%, 1.5%, and 3.3%, respectively, on the MSR-VTT dataset. Therefore, the video captioning method based on semantic topic association can extract more topic information, thereby improving the effectiveness of video caption generation.Keywords
Funding Information
- Chinese Postdoctoral Science Foundation (2020M673446)
This publication has 35 references indexed in Scilit:
- Streaming Dense Video CaptioningarXiv, 2024
- A Video Captioning Method by Semantic Topic-Guided GenerationComputers, Materials & Continua, 2024
- A semi-supervised deep learning image caption model based on Pseudo Label and N-gramInternational Journal of Approximate Reasoning, 2021
- Enhancing the alignment between target words and corresponding frames for video captioningPattern Recognition, 2021
- Video Captioning Based on Channel Soft Attention and Semantic ReconstructorFuture Internet, 2021
- Generating Video Descriptions With Latent Topic GuidanceIEEE Transactions on Multimedia, 2019
- Image Understanding using vision and reasoning through Scene Description GraphComputer Vision and Image Understanding, 2018
- Inception-v4, Inception-ResNet and the Impact of Residual Connections on LearningProceedings of the AAAI Conference on Artificial Intelligence, 2017
- BabyTalk: Understanding and Generating Simple Image DescriptionsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Natural Language Description of Human Activities from Video Images Based on Concept Hierarchy of ActionsInternational Journal of Computer Vision, 2002