TEAdapter: Supply abundant guidance for controllable text-to-music generation

Authors:

JZJialing Zou
Jialing Zou
JMJiahao Mei
Jiahao Mei
XNXudong Nan
Xudong Nan
JLJinghua Li
Jinghua Li
Daoguo Dong
Daoguo Dong
Liang He
Liang He

Preprint

Publisher website

Google Scholar

9 August 2024

preprint
Published by arXiv in arXiv

DOI:10.48550/arXiv.2408.04865

Abstract

Although current text-guided music generation technology can cope with simple creative scenarios, achieving fine-grained control over individual text-modality conditions remains challenging as user demands become more intricate. Accordingly, we introduce the TEAcher Adapter (TEAdapter), a compact plugin designed to guide the generation process with diverse control information provided by users. In addition, we explore the controllable generation of extended music by leveraging TEAdapter control groups trained on data of distinct structural functionalities. In general, we consider controls over global, elemental, and structural levels. Experimental results demonstrate that the proposed TEAdapter enables multiple precise controls and ensures high-quality music generation. Our module is also lightweight and transferable to any diffusion model architecture. Available code and demos will be found soon at https://github.com/Ashley1101/TEAdapter.

Keywords

All Related Versions

Version 1, 2024-08-09, ArXiv

This publication has 10 references indexed in Scilit:

Music ControlNet: Multiple Time-Varying Controls for Music Generation
IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024
AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining
IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024
T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models
Proceedings of the AAAI Conference on Artificial Intelligence, 2024
Adding Conditional Control to Text-to-Image Diffusion Models
2023
Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation
2023
Efficient Large-Scale Audio Tagging Via Transformer-to-CNN Knowledge Distillation
2023
Efficient Training of Audio Transformers with Patchout
2022
High-Resolution Image Synthesis with Latent Diffusion Models
2022
librosa: Audio and Music Signal Analysis in Python
2015
Extracting Predominant Local Pulse Information From Music Recordings
IEEE Transactions on Audio, Speech, and Language Processing, 2010