Reconstructing Speech From CNN Embeddings
- 16 April 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 28 (10709908) , 952-956
Abstract
The complete understanding of the decision-making process of Convolutional Neural Networks (CNNs) is far from being fully reached. Many researchers proposed techniques to interpret what a network actually learns from data. Nevertheless many questions still remain unanswered. In this work we study one aspect of this problem by reconstructing speech from the intermediate embeddings computed by a CNNs. Specifically, we consider a pre-trained network that acts as a feature extractor from speech audio. We investigate the possibility of inverting these features, reconstructing the input signals in a black-box scenario, and quantitatively measure the reconstruction quality by measuring the word-error-rate of an off-the-shelf ASR model.Experiments performed using two different CNN architectures trained for six different classification tasks, show that it is possible to reconstruct time-domain speech signals that preserve the semantic content, whenever the embeddings are extracted before the fully connected layers. Related source code and additional results are available on GitHub https://polimi-ispl.github.io/speechreconstructionembeddings.Keywords
This publication has 14 references indexed in Scilit:
- Vid2speech: Speech reconstruction from silent video2017
- Audio Set: An ontology and human-labeled dataset for audio events2017
- CNN architectures for large-scale audio classification2017
- "Why Should I Trust You?"2016
- Inverting Visual Representations with Convolutional Networks2016
- Understanding deep image representations by inverting them2015
- Librispeech: An ASR corpus based on public domain audio books2015
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- Visualizing and Understanding Convolutional Networks2014
- Signal estimation from modified short-time Fourier transformIEEE Transactions on Acoustics, Speech, and Signal Processing, 1984