Self-Supervised Learning of Multi-Level Audio Representations for Music Segmentation
- 20 March 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 32 (23299290) , 2141-2152
Abstract
The task of music structure analysis refers to automatically identifying the location and the nature of musical sections within a song. In the supervised scenario, structural annotations generally result from exhaustive data collection processes, which represents one of the main challenges of this task. Moreover, both the subjectivity of music structure and the hierarchical characteristics it exhibits make the obtained structural annotations not fully reliable, in the sense that they do not convey a “universal ground-truth” unlike other tasks in music information retrieval. On the other hand, the quickly growing quantity of available music data has enabled weakly supervised and self-supervised approaches to achieve impressive results on a wide range of music-related problems. In this work, a self-supervised learning method is proposed to learn robust multi-level music representations prior to structural segmentation using contrastive learning. To this end, sets of frames sampled at different levels of detail are used to train a deep neural network in a disentangled manner. The proposed method is evaluated on both flat and multi-level segmentation. We show that each distinct sub-region of the output embeddings can efficiently account for structural similarity at their own targeted level of detail, which ultimately improves performance of downstream flat and multi-level segmentation. Finally, complementary experiments are carried out to study how the obtained representations can be further adapted to specific datasets using a supervised fine-tuning objective in order to facilitate structure retrieval in domains where human annotations remain scarce.Keywords
Funding Information
- GENCI-IDRIS by using HPC (2022-AD011013255R1)
This publication has 20 references indexed in Scilit:
- Torchaudio: Building Blocks for Audio and Speech Processing2022
- To Catch A Chorus, Verse, Intro, or Anything Else: Analyzing a Song with Structural Functions2022
- Multimodal Metric Learning for Tag-Based Music Retrieval2021
- A Prototypical Triplet Loss for Cover Detection2020
- Learning to Rank Music Tracks Using Triplet Loss2020
- Disentangled Multidimensional Metric Learning for Music Similarity2020
- Multi-Similarity Loss With General Pair Weighting for Deep Metric Learning2019
- Conditional Similarity Networks2017
- Measuring Structural Similarity in MusicIEEE Transactions on Audio, Speech, and Language Processing, 2011
- Structural Segmentation of Musical Audio by Constrained ClusteringIEEE Transactions on Audio, Speech, and Language Processing, 2008