Music Gesture for Visual Sound Separation
Authors:
- Hang ZhaoHZHang Zhao
- Joshua B. TenenbaumJTJoshua B. Tenenbaum
- Antonio TorralbaATAntonio Torralba
Top Cited Papers
- 1 June 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 10475-10484
Abstract
Recent deep learning approaches have achieved impressive performance on visual sound separation tasks. However, these approaches are mostly built on appearance and optical flow like motion feature representations, which exhibit limited abilities to find the correlations between audio signals and visual points, especially when separating multiple instruments of the same types, such as multiple violins in a scene. To address this, we propose ``Music Gesture," a keypoint-based structured representation to explicitly model the body and finger movements of musicians when they perform music. We first adopt a context-aware graph network to integrate visual semantic context with body dynamics and then apply an audio-visual fusion model to associate body movements with the corresponding audio signals. Experimental results on three music performance datasets show: 1) strong improvements upon benchmark metrics for hetero-musical separation tasks (i.e. different instruments); 2) new ability for effective homo-musical separation for piano, flute, and trumpet duets, which to our best knowledge has never been achieved with alternative methods.Keywords
All Related Versions
This publication has 36 references indexed in Scilit:
- Monoaural Audio Source Separation Using Deep Convolutional Neural Networks2017
- Deep Residual Learning for Image Recognition2016
- Deep clustering: Discriminative embeddings for segmentation and separation2016
- Deep Karaoke: Extracting Vocals from Musical Mixtures Using a Convolutional Deep Neural Network2015
- DevNet: A Deep Event Network for multimedia event detection and evidence recounting2015
- The cocktail party problemCurrent Biology, 2009
- Harmony in Motion2007
- The Cocktail Party ProblemNeural Computation, 2005
- Non-negative matrix factorization for polyphonic music transcription2004
- Voice puppetry1999