Self-Supervised Learning of Context-Aware Pitch Prosody Representations
Preprint
Abstract
In music and speech, meaning is derived at multiple levels of context. Affect, for example, can be inferred both by a short sound token and by sonic patterns over a longer temporal window such as an entire recording. In this letter, we focus on inferring meaning from this dichotomy of contexts. We show how contextual representations of short sung vocal lines can be implicitly learned from fundamental frequency ($F_0$) and thus be used as a meaningful feature space for downstream Music Information Retrieval (MIR) tasks. We propose three self-supervised deep learning paradigms which leverage pseudotask learning of these two levels of context to produce latent representation spaces. We evaluate the usefulness of these representations by embedding unseen pitch contours into each space and conducting downstream classification tasks. Our results show that contextual representation can enhance downstream classification by as much as 15\% as compared to using traditional statistical contour features.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Intonation trajectories within tones in unaccompanied soprano, alto, tenor, bass quartet singingThe Journal of the Acoustical Society of America, 2019
- Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech2018
- The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American EnglishPLOS ONE, 2018
- The expression of emotion in the singing voice: Acoustic patterns in vocal performanceThe Journal of the Acoustical Society of America, 2017
- Frequency Estimation from Waveforms Using Multi-Layered Neural Networks2016
- Melody Extraction From Polyphonic Music Signals Using Pitch Contour CharacteristicsIEEE Transactions on Audio, Speech, and Language Processing, 2012
- Communication of emotions in vocal expression and music performance: Different channels, same code?Psychological Bulletin, 2003