Parameter-Efficient Adaptation of Large Vision—Language Models for Video Memorability Prediction

Authors:
Open Access
Publisher website
Google Scholar
Abstract
The accurate modelling of video memorability, or the intrinsic properties that render a piece of audiovisual content more likely to be remembered, will facilitate the development of automatic systems that are more efficient in retrieving, classifying and generating impactful media. Recent studies have indicated a strong correlation between the visual semantics of video and its memorability. This underscores the importance of developing advanced visual comprehension abilities to enhance model performance. It has been demonstrated that Large Vision–Language Models (LVLMs) demonstrate exceptional proficiency in generalist, high-level semantic comprehension of images and video, due to their extensive multimodal pre-training on a vast scale. This work makes use of the vast generalist knowledge of LVLMs and explores efficient adaptation techniques with a view to utilising them as memorability predictors. In particular, the Quantized Low-Rank Adaptation (QLoRA) technique is employed to fine-tune the Qwen-VL model with memorability-related data extracted from the Memento10k dataset. In light of existing research, we propose a particular methodology that transforms Qwen-VL from a language model to a memorability score regressor. Furthermore, we consider the influence of selecting appropriate LoRA hyperparameters, a design aspect that has been insufficiently studied. We validate the LoRA rank and alpha hyperparameters using 5-Fold Cross-Validation and evaluate our best configuration on the official testing portion of the Memento10k dataset, obtaining a state-of-the-art Spearman Rank Correlation Coefficient (SRCC) of 0.744. Consequently, this work represents a significant advancement in modelling video memorability through high-level semantic understanding.
Funding Information
  • Universidad Politécnica de Madrid (PRE2022-105516, 101071191—HORIZON-EIC-2021-PATHFINDERCHALLENGES-01, PID2020-118112RB-C22, PID2023-150584OB-C21, PID2021-126061OB-C43)
  • Spanish Ministry of Education (PRE2022-105516, 101071191—HORIZON-EIC-2021-PATHFINDERCHALLENGES-01, PID2020-118112RB-C22, PID2023-150584OB-C21, PID2021-126061OB-C43)
  • ASTOUND (PRE2022-105516, 101071191—HORIZON-EIC-2021-PATHFINDERCHALLENGES-01, PID2020-118112RB-C22, PID2023-150584OB-C21, PID2021-126061OB-C43)
  • Ministerio de Ciencia e Innovación (PRE2022-105516, 101071191—HORIZON-EIC-2021-PATHFINDERCHALLENGES-01, PID2020-118112RB-C22, PID2023-150584OB-C21, PID2021-126061OB-C43)
  • Ministerio de Ciencia e Innovación (PRE2022-105516, 101071191—HORIZON-EIC-2021-PATHFINDERCHALLENGES-01, PID2020-118112RB-C22, PID2023-150584OB-C21, PID2021-126061OB-C43)
  • Ministerio de Ciencia e Innovación (PRE2022-105516, 101071191—HORIZON-EIC-2021-PATHFINDERCHALLENGES-01, PID2020-118112RB-C22, PID2023-150584OB-C21, PID2021-126061OB-C43)
  • MCIN/AEI/10.13039/501100011033 (PRE2022-105516, 101071191—HORIZON-EIC-2021-PATHFINDERCHALLENGES-01, PID2020-118112RB-C22, PID2023-150584OB-C21, PID2021-126061OB-C43)
  • European Union “NextGenerationEU/PRTR” (PRE2022-105516, 101071191—HORIZON-EIC-2021-PATHFINDERCHALLENGES-01, PID2020-118112RB-C22, PID2023-150584OB-C21, PID2021-126061OB-C43)

This publication has 30 references indexed in Scilit: