Estefania Cano

This work investigates how personalised Music Emotion Recognition (MER) systems may lead to sensitive profiling when applied to musically induced emotions in politically charged contexts. We focus on traditional Colombian music with explicit political content, including (1) vallenatos and social songs aligned with the left-wing guerrilla Fuerzas Armadas Revolucionarias de Colombia (FARC), and (2) corridos linked to sympathisers of the right-wing paramilitary group Autodefensas Unidas de Colombia (AUC). Using data from 49 participants with diverse political leanings, we train personalised machine learning models to predict induced emotional responses – particularly negative emotions. Our findings reveal that political identity plays a significant role in shaping emotional experiences of music with explicit political content, and that emotion recognition models can capture this variation to a certain extent. These results raise critical concerns about the potential misuse of emotion recognition technologies. What is often framed as a tool for wellbeing and emotional regulation could, in politically sensitive contexts, be repurposed for user profiling. This work highlights the ethical risks of deploying AI-driven emotion analysis without safeguards, particularly among populations that are politically or socially vulnerable. We argue that subjective emotional responses may constitute sensitive personal data, and that failing to account for their sociopolitical context could amplify harm and exclusion.

The COVID-19 pandemic was accompanied by a marked increase in the use of music listening for self-regulation ¹ . During these challenging times, listeners reported they used music ‘to keep them company’ ² ; indicating that they may have turned to music for social solace ³ . However, whether this is simply a figure of speech or an empirically observable effect on social thought was previously unclear. In three experiments, six hundred participants were presented with silence or task-irrelevant music in Italian, Spanish, or Swedish while performing a directed mental-imagery task in which they imagined a journey towards a topographical landmark ⁴ . To control for a possible effect of vocals on imagined content, the music was presented with or without vocals to the participants, of which half were native speakers and the other half non-speakers of the respective languages. Music, compared to silence, led to more vivid imagination and changes in imagined content. Specifically, social interaction emerged as a clear thematic cluster in participants’ descriptions of their imagined content through Latent Dirichlet Allocation. Moreover, Bayesian Mixed effects models revealed that music significantly increased imagined social content compared to silence conditions. This effect remained robust irrespective of vocals or language comprehension. Using stable diffusion, we generated visualisations of participants’ imagined content. In a fourth experiment, a new group of participants was able to use these visualisations to differentiate between content imagined during music listening and that of the silence condition, but only when listening to the associated music. Results converge to show that music, indeed, can be good company.

Including unlabeled data in the training process of neural networks using Semi-Supervised Learning (SSL) has shown impressive results in the image domain, where state-of-the-art results were obtained with only a fraction of the labeled data. The commonality between recent SSL methods is that they strongly rely on the augmentation of unannotated data. This is vastly unexplored for audio data. In this work, SSL using the state-of-the-art FixMatch approach is evaluated on three audio classification tasks, including music, industrial sounds, and acoustic scenes. The performance of FixMatch is compared to Convolutional Neural Networks (CNN) trained from scratch, Transfer Learning, and SSL using the Mean Teacher approach. Additionally, a simple yet effective approach for selecting suitable augmentation methods for FixMatch is introduced. FixMatch with the proposed modifications always outperformed Mean Teacher and the CNNs trained from scratch. For the industrial sounds and music datasets, the CNN baseline performance using the full dataset was reached with less than 5% of the initial training data, demonstrating the potential of recent SSL methods for audio data. Transfer Learning outperformed FixMatch only for the most challenging dataset from acoustic scene classification, showing that there is still room for improvement.

Our previous research showed promising results when transferring features learned from speech to train emotion recognition models for music. In this context, we implemented a denoising autoencoder as a pretraining approach to extract features from speech in two languages (English and Mandarin). From that, we performed transfer and multi-task learning to predict classes from the arousal-valence space of music emotion. We tested and analyzed intra-linguistic and cross-linguistic settings, depending on the language of speech and lyrics of the music. This paper presents additional investigation on our approach, which reveals that: (1) performing pretraining with speech in a mixture of languages yields similar results than for specific languages - the pretraining phase appears not to exploit particular language features, (2) the music in Mandarin dataset consistently results in poor classification performance - we found low agreement in annotations, and (3) novel methodologies for representation learning (Contrastive Predictive Coding) may exploit features from both languages (i.e., pretraining on a mixture of languages) and improve classification of music emotions in both languages. From this study we conclude that more research is still needed to understand what is actually being transferred in these type of contexts.

The goal of this work is to investigate what singing voice separation approaches based on neural networks learn from the data. We examine the mapping functions of neural networks based on the denoising autoencoder (DAE) model that are conditioned on the mixture magnitude spectra. To approximate the mapping functions, we propose an algorithm inspired by the knowledge distillation, denoted the neural couplings algorithm (NCA). The NCA yields a matrix that expresses the mapping of the mixture to the target source magnitude information. Using the NCA, we examine the mapping functions of three fundamental DAE-based models in music source separation; one with single- layer encoder and decoder, one with multi-layer encoder and single-layer decoder, and one using skip-filtering connections (SF) with a single-layer encoding and decoding. We first train these models with realistic data to estimate the singing voice magnitude spectra from the corresponding mixture. We then use the optimized models and test spectral data as input to the NCA. Our experimental findings show that approaches based on the DAE model learn scalar filtering operators, exhibiting a predominant diagonal structure in their corresponding mapping functions, limiting the exploitation of inter-frequency structure of music data. In contrast, skip-filtering connections are shown to assist the DAE model in learning filtering operators that exploit richer inter-frequency structures.

Many people listen to recorded music as part of their everyday lives, e.g., from radio or TV programs, compact discs, downloads, or, increasingly, online streaming services. Sometimes we might want to remix the balance within the music, perhaps to make the vocals louder or to suppress an unwanted sound, or we might want to upmix a two-channel stereo recording to a 5.1-channel surround sound system. We might also want to change the spatial location of a musical instrument within the mix. All of these applications are relatively straightforward, provided we have access to separate sound channels (stems) for each musical audio object.

In this study, we examine the effect of various objective functions used to optimize the recently proposed deep learning architecture for singing voice separation MaD - Masker and Denoiser. The parameters of the MaD architecture are optimized using an objective function that contains a reconstruction criterion between predicted and true magnitude spectra of the singing voice, and a regularization term. We examine various reconstruction criteria such as the generalized Kullback-Leibler, mean squared error, and noise to mask ratio. We also explore recently proposed, for optimizing MaD, regularization terms such as sparsity and TwinNetwork regularization. Results from both objective assessment and listening tests suggest that the TwinNetwork regularization results in improved singing voice separation quality.

Quality of sound source separation algorithms has traditionally been evaluated based on a set of established quality metrics: target distortion, interference from other sources, artifacts distortion, and a measure of overall separation quality. In our previous work, listening test results were presented where no significant correlation between these quality metrics and perceptual ratings from the listening test could be observed. Following these results, we now attempt to better understand perceptual quality in a sound source separation context. We focus on determining how separation quality is actually defined by listeners and propose the use of a descriptive methodology to reveal its most relevant dimensions. A combination of Free-Choice Profiling and Repertory Grid Technique is used with 10 human listeners in an attempt to verify if the main dimensions of separation quality truly correspond to those established by the quality metrics. The outcomes of this exploration bring light to the development of new methodologies for sound separation quality evaluation, and suggest a two-dimensional perceptual space for quality of sound source separation.