SUMMR: A Unified Multimodal Representation Framework for Songs

Authors:
Publisher website
Google Scholar
Abstract
The understanding and representation of songs is a crucial issue in music platforms, as it can facilitate numerous applications in the music field. Songs are a common multimodal art form within the music domain, achieving rich musical connotations and strong expressiveness. However, the data of songs exhibit obvious multimodal and heterogeneous characteristics, presenting significant challenges to the understanding and representation of songs. Regrettably, the current methods do not respond to these challenges effectively. To this end, in this study, a unified multimodal representation framework for songs, namely SUMMR, is proposed. Specifically, first, a two-layer framework is put forward. In embedding layer, the features of different modalities data are embedded into a unified space. In content layer, a novel cross-modal attention mechanism is designed, which effectively capture the cross-modal semantic associations and deep music features, thereby obtaining a unified representation of songs. Then, a two-level hierarchical pre-training algorithm is proposed, which can effectively lower the training cost. Finally, experiments are conducted on two typical music tasks with public datasets of songs, where the experimental results demonstrate the effectiveness of SUMMR for understanding and representation of songs, and also show that SUMMR has good capability of being fine-tuned in many song-based tasks.
Funding Information
  • grants from the University synergy innovanon Program of Anhui provinc (GXXT-2022-042)
  • grants from the University synergy innovanon Program of Anhui provinc (GXXT-2022-101)
  • Open Research Fund of Key Laboratory of philosophyand Social science of Anhui Provinc (SYS2023A06)
  • National Natural Science Foundation of China (sU20A20229)
  • he Key Projects of the National Natural Science Foundation of Universities in Anhui Provinc (2023AH051302)
  • The Scientific Research Foundation of Hefei Normal Universit (2022rcjj57)
  • The Open Project of Anhui Provincial Key Laboratory of Multimodal Cognitive Computation, Anhui Universit (MMC202405)