MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 13647-13657
Abstract
We present MM-Narrator, a novel system leveraging GPT-4 with multimodal in-context learning for the gener-ation of audio descriptions (AD). Unlike previous methods that primarily focused on downstream fine-tuning with short video clips, MM-Narrator excels in generating precise audio descriptions for videos of extensive lengths, even be-yond hours, in an autoregressive manner. This capability is made possible by the proposed memory-augmented generation process, which effectively utilizes both the short-term textual context and long-term visual memory through an efficient register-and-recall mechanism. These contextual memories compile pertinent past information, including storylines and character identities, ensuring an accurate tracking and depicting of story-coherent and character-centric audio descriptions. Maintaining the training-free design of MM-Narrator, we further propose a complexity-based demonstration selection strategy to largely enhance its multi-step reasoning capability via few-shot multimodal in-context learning (MM-ICL). Experimental results on MAD-eval dataset demonstrate that MM-Narrator consistently outperforms both the existing fine-tuning-based approaches and LLM-based approaches in most scenarios, as measured by standard evaluation metrics. Additionally, we introduce the first segment-based evaluator for recurrent text generation. Empowered by GPT-4, this evaluator comprehensively reasons and marks AD generation performance in various extendable dimensions.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- AutoAD: Movie Description in Context2023
- YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors2023
- SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning2022
- MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptions2022
- Movie DescriptionInternational Journal of Computer Vision, 2017
- Gains and losses of watching audio described films for sighted viewersTarget. International Journal of Translation Studies, 2016
- SPICE: Semantic Propositional Image Caption Evaluation2016
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language2016
- CIDEr: Consensus-based image description evaluation2015
- A dataset for Movie Description2015