A Context Encoder For Audio Inpainting
- 14 October 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 27 (12) , 2362-2372
Abstract
We study the ability of deep neural networks (DNNs) to restore missing audio content based on its context, i.e., inpaint audio gaps. We focus on a condition which has not received much attention yet: gaps in the range of tens of milliseconds. We propose a DNN structure that is provided with the signal surrounding the gap in the form of time-frequency (TF) coefficients. Two DNNs with either complex-valued TF coefficient output or magnitude TF coefficient output were studied by separately training them on inpainting two types of audio signals (music and musical instruments) having 64-ms long gaps. The magnitude DNN outperformed the complex-valued DNN in terms of signal-to-noise ratios and objective difference grades. Although, for instruments, a reference inpainting obtained through linear predictive coding performed better in both metrics, it performed worse than the magnitude DNN for music. This demonstrates the potential of the magnitude DNN, in particular for inpainting signals that are more complex than single instrument sounds.Keywords
All Related Versions
Funding Information
- Austrian Science Fund (I 3067-N30)
- NVIDIA Corporation
This publication has 37 references indexed in Scilit:
- Sparsity and Cosparsity for Audio Declipping: A Flexible Non-convex Approach2015
- Joint Audio Inpainting and Source Separation2015
- Self-content-based audio inpaintingSignal Processing, 2015
- Computational auditory induction as a missing-data model-fitting problem with Bregman divergenceSpeech Communication, 2010
- Missing Data Imputation for Time-Frequency Representations of Audio SignalsJournal of Signal Processing Systems, 2010
- Restoration of a discrete-time signal segment by interpolation based on the left-sided and right-sided autoregressive parametersIEEE Transactions on Signal Processing, 1996
- Waveform substitution techniques for recovering missing speech segments in packet voice communicationsIEEE Transactions on Acoustics, Speech, and Signal Processing, 1986
- Adaptive interpolation of discrete-time signals that can be modeled as autoregressive processesIEEE Transactions on Acoustics, Speech, and Signal Processing, 1986
- Signal estimation from modified short-time Fourier transformIEEE Transactions on Acoustics, Speech, and Signal Processing, 1984
- Implementation of the digital phase vocoder using the fast Fourier transformIEEE Transactions on Acoustics, Speech, and Signal Processing, 1976