Double Linear Transformer for Background Music Generation from Videos
Open Access
- 17 May 2022
- journal article
- research article
- Published by MDPI AG in Applied Sciences
- Vol. 12 (10) , 5050
Abstract
Many music generation research works have achieved effective performance, while rarely combining music with given videos. We propose a model with two linear Transformers to generate background music according to a given video. To enhance the melodic quality of the generated music, we firstly input note-related and rhythm-related music features separately into each Transformer network. In particular, we pay attention to the connection and the independence of music features. Then, in order to generate the music that matches the given video, the current state-of-the-art cross-modal inference method is set up to establish the relationship between visual mode and sound mode. Subjective and objective experiment indicate that the generated background music matches the video well and is also melodious.Keywords
Funding Information
- National Key R&D Program of China (No. 2021YFF0900701)
This publication has 15 references indexed in Scilit:
- Chord-Conditioned Melody Harmonization With Controllable Harmonicity2023
- Rethinking Semantic Segmentation: A Prototype View2022
- Foley Music: Learning to Generate Music from Videos2020
- PopMAG2020
- Pop Music Transformer2020
- EvoComposer: An Evolutionary Algorithm for 4-Voice Music CompositionsEvolutionary Computation, 2020
- This time with feeling: learning expressive musical performanceNeural Computing and Applications, 2018
- DeepJ: Style-Specific Music Generation2018
- Deep Cross-Modal Audio-Visual Generation2017
- MetaCompose: A Compositional Evolutionary Music Composer2016