Sergio Esteban-Romero

Emotion recognition from bodily movement remains a challenging problem, particularly when only pose-based motion sequences are available and emotionally informative content is not uniformly distributed across time. In this work, we propose a Window Transformer architecture grounded in the Multiple Instance Learning (MIL) paradigm to address this challenge. Rather than processing the full sequence as a single temporal stream, the model decomposes it into overlapping windows and learns to assign greater relevance to those segments containing stronger emotional content. This formulation provides a more interpretable framework, since the learned relevance scores reveal which temporal regions drive the final prediction, while also yielding richer, context-aware representations of each segment. We evaluate the proposed approach on two publicly available datasets, MEED and DIEM-A, and compare it against a standalone Transformer baseline under different batch size and window configuration settings. The Window Transformer consistently outperforms the baseline and exhibits a more stable behavior across training configurations, achieving best accuracies of 56.35 ± 2.67% on MEED and 23.84 ± 0.83% on DIEM-A in a subject-independent scenario. Beyond performance gains, this work also establishes new benchmark values on both datasets, providing reference results that future research on bodily emotion recognition can build on.

Isolated Sign Language Recognition (ISLR) systems are increasingly expected to operate under real-world constraints, where video data exhibit high variability and practical deployments demand low latency on consumer-grade hardware. While many recent approaches rely on pose or skeleton representations to reduce input complexity, such pipelines may discard fine-grained visual cues that are critical for accurate recognition. In this work, we investigate an end-to-end RGB formulation for ISLR based on fine-tuning the V-JEPA2 video transformer, complemented by a lightweight frame-selection pipeline designed to reduce computational cost while preserving linguistically informative temporal segments. Our approach integrates motion-aware sampling strategies and confidence-guided filtering of hand keypoints to mitigate common issues such as hands-down segments or partial hand visibility. We evaluate the proposed method on two complementary benchmarks: AVASAG100, recorded in a controlled transportation-oriented environment, and WLASL100, a challenging in-the-wild dataset characterized by substantial signer and environmental variability. Despite operating in a limited-data downstream setting and using a compact parameter budget, our model achieves performance comparable to current state-of-the-art methods, reaching 98.09±0.90% accuracy on AVASAG100 and 87.98±3.97% on WLASL100, without relying on additional sign-language-specific pretraining. These results support the viability of efficient, end-to-end video-transformer-based ISLR for domain-specific and resource-constrained applications.

Video memorability prediction has emerged as a key challenge for improving information retrieval, content design, and user engagement. Prior work has shown that semantic cues play a crucial role in determining memorability, with recent studies leveraging Contrastive Language-Image Pre-training (CLIP) encoders to incorporate semantic information. However, the specific improvements attributable to CLIP models remain unclear, as few studies systematically compare their performance against equivalent unimodal encoders or explore fine-tuning strategies. This work addresses that gap through a comprehensive, controlled evaluation of CLIP-based and unimodal encoders for video memorability prediction. We propose FCLIP, a domain-adapted extension of CLIP that undergoes additional contrastive pre-training on memorability-specific image-text pairs. Our experiments assess both feature extraction and supervised fine-tuning, ensuring fair comparisons across models with matched architecture and parameter count. Results show that FCLIP image encoders achieve a Spearman Rank Correlation Coefficient (SRCC) of 0.672 on the Memento10k dataset, significantly outperforming unimodal Vision Transformers. FCLIP text encoders similarly outperform unimodal baselines, reaching an SRCC of 0.632. These findings demonstrate that contrastive learning and domain adaptation substantially improve memorability prediction, highlighting the importance of semantic and multimodal pre-training in developing advanced content analysis systems.

Sign Languages are the primary communication modality of deaf communities, yet building effective Isolated Sign Language Recognition (ISLR) systems remains difficult under data limitations. In this work, we curated a sub-dataset from the DGS-Korpus focused on recognizing affirmations and negations (polar answers) in German Sign Language (DGS). We designed lightweight transformer models using landmark-based inputs and evaluated them on two tasks: the binary classification of affirmations versus negations (binary semantic recognition) and the multi-class recognition of sign variations expressing positive or negative replies (multi-class gloss recognition). The main contribution of the article, hence, relies on the exploration of models for performing polar answer recognition in DGS and the exploration of differences between performing multi-class or binary class classification. Our best binary model achieved an accuracy of 97.71% using only hand landmarks without Positional Encoding, highlighting the potential of lightweight landmark-based transformers for efficient ISLR in constrained domains.

Olfactory information, crucial for human perception, is often underrepresented compared to visual and textual data. This work explores methods for understanding smell descriptions within a multimodal context, where scent information is conveyed indirectly through text and images. We address the challenges of the Multimodal Understanding of Smells in Texts and Images (MUSTI) task by proposing novel approaches that leverage language-specific models and state-of-the-art multimodal large language models (MM-LLMs). Our core contribution is a multimodal framework using language-specific encoders for text and image data. This allows for a joint embedding space that explores the semantic symmetry between smells, texts, and images to identify olfactory-related connections shared across the modalities. While ensemble learning with language-specific models achieved good performance, MM-LLMs demonstrated exceptional potential. Fine-tuning a quantized version of the Qwen-VL-Chat model achieved a state-of-the-art macro F1-score of 0.7618 on the MUSTI task. This highlights the effectiveness of MM-LLMs in capturing task requirements and adapting to specific formats.

The accurate modelling of video memorability, or the intrinsic properties that render a piece of audiovisual content more likely to be remembered, will facilitate the development of automatic systems that are more efficient in retrieving, classifying and generating impactful media. Recent studies have indicated a strong correlation between the visual semantics of video and its memorability. This underscores the importance of developing advanced visual comprehension abilities to enhance model performance. It has been demonstrated that Large Vision–Language Models (LVLMs) demonstrate exceptional proficiency in generalist, high-level semantic comprehension of images and video, due to their extensive multimodal pre-training on a vast scale. This work makes use of the vast generalist knowledge of LVLMs and explores efficient adaptation techniques with a view to utilising them as memorability predictors. In particular, the Quantized Low-Rank Adaptation (QLoRA) technique is employed to fine-tune the Qwen-VL model with memorability-related data extracted from the Memento10k dataset. In light of existing research, we propose a particular methodology that transforms Qwen-VL from a language model to a memorability score regressor. Furthermore, we consider the influence of selecting appropriate LoRA hyperparameters, a design aspect that has been insufficiently studied. We validate the LoRA rank and alpha hyperparameters using 5-Fold Cross-Validation and evaluate our best configuration on the official testing portion of the Memento10k dataset, obtaining a state-of-the-art Spearman Rank Correlation Coefficient (SRCC) of 0.744. Consequently, this work represents a significant advancement in modelling video memorability through high-level semantic understanding.

Human gesture recognition using electromyography (EMG) signals holds high potential for enhancing the functionality of human–machine interfaces, prosthetic devices, and sports performance analysis. This work proposes a gesture classification system based on electromyography. This system has been designed to improve the accuracy of forearm gesture classification by leveraging advanced signal processing and deep learning techniques to optimize classification accuracy. The system is composed of two main modules: a signal processing module, able to perform two main transforms (short-time Fourier transform and Constant-Q-Transform), and a classification module based on convolutional neural networks (CNNs). The dataset employed in this study, entitled “Latent Factors Limiting the Performance of sEMG-Interfaces”, comprises EMG signals collected via a bracelet equipped with 8 distinct sensors, capable of capturing a wide range of forearm muscle activities. The experimental process is composed of two main phases. Firstly, we employed a k-fold cross-validation methodology to systematically assess and validate the model’s performance across different subsets of data for hyperparameter tunning. Secondly, the best system configuration was evaluated using a new subset, reporting significant improvements. The baseline neural network architecture reported an accuracy of 85.0 ± 0.13% when classifying gestures. Through rigorous hyperparameter tuning and the application of various mathematical transformations to the EMG features, we managed to enhance the classification accuracy to 90.0 ± 0.12% (an absolute improvement of 5% compared to the baseline for a 5-class problem). When making comparisons to previous works, we significantly improved the F-score from 85.5% to 89.3% for a 4-class problem (left, right, up, and down).

This paper describes the development of a Human Activity Recognition (HAR) system based on deep learning for classifying full-body activities using inertial signals. The HAR system is divided into several modules: a preprocessing module for extracting relevant features from the inertial signals window-by-window, a machine learning algorithm for classifying the windows and a post-processing module for integrating the information along several windows. Regarding the preprocessing module, several transformations are implemented and evaluated. For the ML module, several algorithms are evaluated, including several deep learning architectures. This evaluation has been carried out over the HARTH dataset. This public dataset contains recordings from 22 participants wearing two 3-axial Axivity AX3 accelerometers for 2 h in a free-living setting. Not all the subjects completed the whole session. Sixteen different activities were recorded and annotated accordingly. This paper describes the fine-tuning process of several machine learning algorithms and analyses their performance with different sets of activities. The best results show an accuracy of 90% and 93% for 12 and nine activities, respectively. To the author’s knowledge, these analyses provide the best state-of-the-art results over this public dataset. Additionally, this paper includes several analyses of the confusion between the different activities.