Iván Martín-Fernández

Advances in Artificial Intelligence (AI) and research on human cognition have long evolved in parallel, each informing and constraining the other. This dissertation adopts that perspective and treats AI as a scientific instrument for studying subjective human perception. It asks whether inherently subjective perceptual phenomena still exhibit enough regularity to be modeled computationally, provided that semantic structure, multimodal representations, and principled adaptation are used. The objective of this thesis is to develop and critically evaluate systems that predict subjective human perception variables from multimedia content based on semantically grounded, large-scale multimodal models under task-faithful protocols, using parameter-efficient adaptation. This objective is firstly grounded by an initial case study on Electroencephalography (EEG)-based movie recall prediction and then instantiated across video memorability prediction, persuasion strategy detection, and social perception modeling. To this end, the dissertation first proposes a high level theoretical framework of subjective perception grounded in a cognitive loop model that links sensory input, attention, memory, internal representations, and action. This framework distinguishes between perceptual judgments with high intersubject stability and more subjective variables that depend systematically on observer and context. Empirically, large pre-trained multimodal models are treated as strong semantic priors and evaluated under protocols that account for ambiguity and variability in subjective annotations. An initial EEG-based case study shows that functionally grounded oscillatory descriptors contain predictive information about movie recall despite substantial inter-subject variability. In video memorability prediction, results reveal structured interactions between semantic interpretation, visual processing, and temporal dynamics, but also limited cross setup generalization. Contrastive adaptation and low-rank adaptation of large vision-language models often obtain stronger prediction results under the evaluated protocols. However, scaling effects are not monotonic, suggesting that alignment, backbone design, and adaptation capacity interact more strongly than total model size alone. In persuasion strategy detection, reformulating evaluation under a unified multilabel framework exposes limitations of prior protocols and shows that zero shot and adapted multimodal models can achieve competitive performance, while attention and error analyses reveal task specific redistribution patterns and persistent cross dataset fragility. In social perception modeling, results indicate the presence of a latent structure organizing social traits, while also suggesting that simple probing or direct low-rank adaptation of frozen models is not, by itself, sufficient in this low-resource setting. Stronger performance emerges from architectures that project pre-trained audiovisual features into a shared semantic space processed by a largely frozen language backbone. Overall, the thesis argues that subjective human perception is neither arbitrary nor trivially predictable. It exhibits structured regularities that can be captured when models are semantically grounded, adapted in a principled manner, and evaluated with robust and task-faithful criteria. At the same time, generalization across domains and setups remains fragile, constraining deployment. The key outcome is a set of empirical findings and methodological guardrails for studying subjective perception with pre-trained multimodal models, including where robustness breaks. These findings support the future development of tools for media auditing, high-quality content creation, and more informed access to multimedia content. RESUMEN Los avances en Inteligencia Artificial (IA) y la investigación sobre la cognición humana han evolucionado en paralelo durante décadas, informándose y limitándose mutuamente. Esta tesis adopta esa perspectiva y concibe la IA como instrumento científico para estudiar la percepción humana subjetiva. En concreto, investiga si fenómenos perceptivos intrínsecamente subjetivos presentan regularidades suficientes para ser modelados computacionalmente cuando se emplean estructuras semánticas, representaciones multimodales y estrategias de adaptación fundamentadas. El objetivo es desarrollar y evaluar críticamente modelos multimodales de gran escala, semánticamente fundamentados, capaces de predecir variables de percepción subjetiva a partir de contenido multimedia bajo protocolos fieles a la tarea, mediante su adaptación eficiente. Este objetivo se fundamenta en primer lugar mediante un estudio de caso sobre predicción de recuerdo de vídeos a partir de electroencefalografía (EEG), y posteriormente se concreta en tres dominios: predicción de la memorabilidad de un vídeo, detección de estrategias de persuasión y modelado de la percepción social. Para ello, la tesis propone un marco teórico sobre la percepción subjetiva basado en un modelo cognitivo que conecta entrada sensorial, atención, memoria, representaciones internas y acción. Distingue entre variables objetivas, con alta estabilidad intersujeto, y subjetivas, que dependen del observador y el contexto. Un estudio de caso inicial basado en EEG muestra que descriptores oscilatorios fundamentados funcionalmente contienen información predictiva sobre el recuerdo de vídeos, a pesar de la considerable variabilidad intersujeto. Empíricamente, los modelos multimodales preentrenados se tratan como fuertes priors semánticos y se evalúan con protocolos que consideran la ambigüedad y variabilidad de las anotaciones. En memorabilidad, los resultados revelan interacciones entre interpretación semántica, procesamiento visual y dinámica...

This paper introduces TESSCCo (TV-control EEG-based Silent Speech Command Corpus), a new dataset including electroencephalography (EEG) signals during Overt Speech (OS) and Covert Speech (CS) in different languages. The dataset comprises repetitions of five different commands pronounced covertly and overtly in English and Spanish from 21 healthy native Spanish speakers (13 male, 8 female, 23 ± 2 years old), while EEG and audio were recorded. In addition, 3 non-native healthy Spanish speakers were recorded under the same circumstances (3 male, 23.3 ± 0.6 years old). A total of 7936 available epochs (i.e., 11.02 hours of data) were recorded with a 32 channel, 256 Hz sampling rate Water based EEG device. The database was designed to maximize the number of different analysis involving EEG signals. The final number of epochs, as well as the statistical analysis (showing significance in Broca's and Wernicke's areas) and machine learning experiments (with subjects exceeding the chance level with basic machine learning models), show that this material is a valuable resource for research on future ways of communication.

The automatic detection of persuasive strategies in advertisements presents a uniquely multimodal challenge at the intersection of vision, language, and social cognition. While recent advances in Large Vision-Language Models (LVLMs) offer promising capabilities for such tasks, current approaches often rely on restrictive evaluation schemes that do not reflect the inherently multi-label nature of persuasive messaging. In this work, we reframe persuasion strategy detection as a genuine multi-label classification problem and propose a principled evaluation framework to enhance interpretability and robustness. We apply this approach to both image and video datasets, examine their characteristics, and introduce novel input-agnostic baselines that achieve macro F1 scores of 0.082 and 0.289 on the respective test sets. As part of this analysis, we study label co-occurrence patterns and dataset ambiguities, providing insights that inform both model interpretation and future dataset design. To assess the native capabilities of LVLMs, we benchmark three open source models—PaliGemma, PaliGemma2, and Qwen2.5-VL—on the image persuasion dataset. Under zero-shot conditions, we demonstrate that querying each strategy individually with a logit-based decision threshold outperforms guided text generation. The best-performing zero-shot model, Qwen2.5-VL, achieves a macro F1 score of 0.227 and a sample F1 score of 0.234 on the image test set, and 0.381 and 0.400 respectively on the video test set. We further explore and compare two lightweight fine-tuning strategies that update only small subsets of model parameters while keeping the remaining weights frozen: fine-tuning of the image-to-text tokens linear projection and Low Rank Adaptation (LoRA) of the language model. Linear projector fine-tuning yields a top macro F1 score of 0.396 on the image test set, marking a substantial improvement over zero-shot performance. To evaluate cross-modal generalization, we apply fine-tuned image models to the video dataset. Our experiments reveal that, while projection-based fine-tuning enables partial knowledge transfer from image to video (macro F1 = 0.416 on a testing subset), LoRA adaptation severely disrupts cross-modal performance (macro F1 = 0.189). Finally, we perform a per-strategy performance analysis, looking into annotator- and data-centric factors that may influence LVLM performance. These findings highlight the viability of open-weight LVLMs for fine-grained persuasion analysis and suggest efficient pathways for domain-specific adaptation under realistic resource constraints.

One of the powers of visual media lies in its ability to create a lasting impression on the viewer’s memory. In this digital age, where media is abundant and attention spans are fleeting, the task of predicting which content will stick in the viewer’s mind has become a critical challenge in computer vision. Computational memorability seeks to address this by developing models that estimate how memorable a piece of media is likely to be. In this review we focus on the MediaEval Predicting Video Memorability benchmark, a recurring evaluation task that has run annually since 2018. This benchmark provides a unique and consistent framework for researchers to compare and refine their memorability prediction techniques using standardised datasets and metrics. Its reproducible framework has proven invaluable for tracking progress and fostering innovation in this rapidly evolving domain. We analyse the evolution of the benchmark across its 2018–2023 editions, discussing the challenges that still remain, such as the need for more interpretability in models and the difficulty of predicting subjective and context-dependent memorability. By analysing and synthesising the collective insights gained from this task, we endeavour to inspire new avenues of inquiry and drive progress towards a more comprehensive understanding of this topic.

Isolated Sign Language Recognition (ISLR) systems are increasingly expected to operate under real-world constraints, where video data exhibit high variability and practical deployments demand low latency on consumer-grade hardware. While many recent approaches rely on pose or skeleton representations to reduce input complexity, such pipelines may discard fine-grained visual cues that are critical for accurate recognition. In this work, we investigate an end-to-end RGB formulation for ISLR based on fine-tuning the V-JEPA2 video transformer, complemented by a lightweight frame-selection pipeline designed to reduce computational cost while preserving linguistically informative temporal segments. Our approach integrates motion-aware sampling strategies and confidence-guided filtering of hand keypoints to mitigate common issues such as hands-down segments or partial hand visibility. We evaluate the proposed method on two complementary benchmarks: AVASAG100, recorded in a controlled transportation-oriented environment, and WLASL100, a challenging in-the-wild dataset characterized by substantial signer and environmental variability. Despite operating in a limited-data downstream setting and using a compact parameter budget, our model achieves performance comparable to current state-of-the-art methods, reaching 98.09±0.90% accuracy on AVASAG100 and 87.98±3.97% on WLASL100, without relying on additional sign-language-specific pretraining. These results support the viability of efficient, end-to-end video-transformer-based ISLR for domain-specific and resource-constrained applications.

Video memorability prediction has emerged as a key challenge for improving information retrieval, content design, and user engagement. Prior work has shown that semantic cues play a crucial role in determining memorability, with recent studies leveraging Contrastive Language-Image Pre-training (CLIP) encoders to incorporate semantic information. However, the specific improvements attributable to CLIP models remain unclear, as few studies systematically compare their performance against equivalent unimodal encoders or explore fine-tuning strategies. This work addresses that gap through a comprehensive, controlled evaluation of CLIP-based and unimodal encoders for video memorability prediction. We propose FCLIP, a domain-adapted extension of CLIP that undergoes additional contrastive pre-training on memorability-specific image-text pairs. Our experiments assess both feature extraction and supervised fine-tuning, ensuring fair comparisons across models with matched architecture and parameter count. Results show that FCLIP image encoders achieve a Spearman Rank Correlation Coefficient (SRCC) of 0.672 on the Memento10k dataset, significantly outperforming unimodal Vision Transformers. FCLIP text encoders similarly outperform unimodal baselines, reaching an SRCC of 0.632. These findings demonstrate that contrastive learning and domain adaptation substantially improve memorability prediction, highlighting the importance of semantic and multimodal pre-training in developing advanced content analysis systems.

Olfactory information, crucial for human perception, is often underrepresented compared to visual and textual data. This work explores methods for understanding smell descriptions within a multimodal context, where scent information is conveyed indirectly through text and images. We address the challenges of the Multimodal Understanding of Smells in Texts and Images (MUSTI) task by proposing novel approaches that leverage language-specific models and state-of-the-art multimodal large language models (MM-LLMs). Our core contribution is a multimodal framework using language-specific encoders for text and image data. This allows for a joint embedding space that explores the semantic symmetry between smells, texts, and images to identify olfactory-related connections shared across the modalities. While ensemble learning with language-specific models achieved good performance, MM-LLMs demonstrated exceptional potential. Fine-tuning a quantized version of the Qwen-VL-Chat model achieved a state-of-the-art macro F1-score of 0.7618 on the MUSTI task. This highlights the effectiveness of MM-LLMs in capturing task requirements and adapting to specific formats.

The accurate modelling of video memorability, or the intrinsic properties that render a piece of audiovisual content more likely to be remembered, will facilitate the development of automatic systems that are more efficient in retrieving, classifying and generating impactful media. Recent studies have indicated a strong correlation between the visual semantics of video and its memorability. This underscores the importance of developing advanced visual comprehension abilities to enhance model performance. It has been demonstrated that Large Vision–Language Models (LVLMs) demonstrate exceptional proficiency in generalist, high-level semantic comprehension of images and video, due to their extensive multimodal pre-training on a vast scale. This work makes use of the vast generalist knowledge of LVLMs and explores efficient adaptation techniques with a view to utilising them as memorability predictors. In particular, the Quantized Low-Rank Adaptation (QLoRA) technique is employed to fine-tune the Qwen-VL model with memorability-related data extracted from the Memento10k dataset. In light of existing research, we propose a particular methodology that transforms Qwen-VL from a language model to a memorability score regressor. Furthermore, we consider the influence of selecting appropriate LoRA hyperparameters, a design aspect that has been insufficiently studied. We validate the LoRA rank and alpha hyperparameters using 5-Fold Cross-Validation and evaluate our best configuration on the official testing portion of the Memento10k dataset, obtaining a state-of-the-art Spearman Rank Correlation Coefficient (SRCC) of 0.744. Consequently, this work represents a significant advancement in modelling video memorability through high-level semantic understanding.

A detailed analysis of the inertial signals input is required when using deep learning models for Parkinson’s Disease detection. This work explores the possibility of reducing the input size of the models by studying the most appropriate frequency range and determines if it is feasible to evaluate subjects with different sensor locations than those used during training. For experimentation, 3.2 s windows are used to classify signals between Parkinson’s patients and control subjects, applying Fast Fourier Transform to the inertial signals and following a Leave-One-Subject-Out Cross-Validation methodology for the PD-BioStampRC21 dataset. It has been observed that the frequency range of 0 to 5 Hz offers a classification accuracy rate of 75.75 ± 0.62% using the five available sensors for training and evaluation, which is close to the model’s performance over the entire frequency range, from 0 to 15.625 Hz, which is 77.46 ± 0.60%. Regarding the transfer learning between sensors located in different body parts, it was observed that training and evaluating the model using data from the right forearm resulted in an accuracy of 65.17 ± 0.69%. When the model was trained with data from the opposite forearm, the accuracy was similar, at 63.57 ± 0.69%. Likewise, comparable results were found when using data from the other forearm and when training and evaluating with opposite thighs, with accuracy reductions not exceeding 3%.