$\mathrm{W}^{2}$: V ariable-Length V ideo-to- M usic Generation

Authors:
Publisher website
Google Scholar
Abstract
The composition of soundtracks for videos is a highly integrated process that synthesizes artistic creativity with technical expertise. It involves the precise synchronization of musical elements with the visual component to effectively enhance the artistic quality of the video. Herein, we aimed to generate harmonious and temporally aligned music based on video content, assisting artists in soundtrack composition. Existing approaches predominantly generate music based on human motion data from fixed-length videos, limiting their applicability. To overcome this limitation, we propose an autoregressive unit diffusion model, $\mathbf {W^{2}}$, for variable-length video-to-music generation. $\mathbf {W^{2}}$ segments videos and music into unified temporal units and employs a diffusion model to generate music units conditioned on structured video information and contextual music information. It then autoregressively generates coherent and consistent music sequences. This model employs structured video condition information, obtained through global encoding, to effectively generate music units that consistently align with the video content. At the same time, contextual music condition information is used to ensure the generation of a continuous sequence of music units matching the duration of the video. Unlike prior models relying on human skeletal features, $\mathbf {W^{2}}$ can be applied to not only dance videos but also free-style videos. Experimental results show that $\mathbf {W^{2}}$ generates music that is temporally synchronized and semantically aligned with visual content.

This publication has 35 references indexed in Scilit: