CLAPSep: Leveraging Contrastive Pre-Trained Model for Multi-Modal Query-Conditioned Target Sound Extraction
- 13 November 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 32 (23299290) , 4945-4960
Abstract
Universal sound separation (USS) aims to extract arbitrary types of sounds from real-world recordings. This can be achieved by language-queried target sound extraction (TSE), which typically consists of two components: a query network that converts user queries into conditional embeddings, and a separation network that extracts the target sound accordingly. Existing methods commonly train models from scratch. As a consequence, substantial data and computational resources are required to make the randomly initialized model comprehend sound events and perform separation accordingly. In this paper, we propose to integrate pre-trained models into TSE models to address the above issue. To be specific, we tailor and adapt the powerful contrastive language-audio pre-trained model (CLAP) for USS, denoted as CLAPSep. CLAPSep also accepts flexible user inputs, taking both positive and negative user prompts of uni- and/or multi-modalities for target sound extraction. These key features of CLAPSep can not only enhance the extraction performance but also improve the versatility of its application. We provide extensive experiments on 5 diverse datasets to demonstrate the superior performance and zero- and few-shot generalizability of our proposed CLAPSep with fast training convergence, surpassing previous methods by a significant margin. Full codes and some audio examples are released for reproduction and evaluationKeywords
All Related Versions
Funding Information
- National Key Research and Development Program of China (2022YFC3302800)
- Innovation and Development Joint Funds of Shandong Natural Science Foundation (ZR2022LZH012)
- Natural Science Foundation of Shandong Province (ZR2023QF103)
This publication has 37 references indexed in Scilit:
- Extract Free Dense Labels from CLIP2022
- Image Segmentation Using Text and Image Prompts2022
- DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting2022
- HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection2022
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
- Attention Is All You Need In Speech Separation2021
- The Sound of Pixels2018
- Supervised Speech Separation Based on Deep Learning: An OverviewIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2018
- An Overview of Lead and Accompaniment Separation in MusicIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2018
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015