Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speech
Authors:
- 4 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-5
Abstract
Recent self-supervised learning (SSL) models have proven to learn rich representations of speech, which can readily be utilized by diverse downstream tasks. To understand such utilities, various analyses have been done for speech SSL models to reveal which and how information is encoded in the learned representations. Although the scope of previous analyses is extensive in acoustic, phonetic, and semantic perspectives, the physical grounding by speech production has not yet received full attention. To bridge this gap, we conduct a comprehensive analysis to link speech representations to articulatory trajectories measured by electromagnetic articulography (EMA). Our analysis is based on a linear probing approach where we measure articulatory score as an average correlation of linear mapping to EMA. We analyze a set of SSL models selected from the leaderboard of the SUPERB benchmark [1] and perform further layer-wise analyses on two most successful models, Wav2Vec 2.0 [2] and HuBERT [3]. Surprisingly, representations from the recent speech SSL models are highly correlated with EMA traces (best: r =0.81), and only 5 minutes are sufficient to train a linear model with high performance (r =0.77). Our findings suggest that SSL models learn to align closely with continuous articulations, and provide a novel insight into speech SSL.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Layer-Wise Analysis of a Self-Supervised Speech Representation Model2021
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden UnitsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- SUPERB: Speech Processing Universal PERformance Benchmark2021
- TERA: Self-Supervised Learning of Transformer Encoder Representation for SpeechIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- Probing Acoustic Representations for Phonetic Properties2021
- Libri-Light: A Benchmark for ASR with Limited or No Supervision2020
- wav2vec: Unsupervised Pre-Training for Speech Recognition2019
- Encoding of Articulatory Kinematic Trajectories in Human Speech Sensorimotor CortexNeuron, 2018
- Librispeech: An ASR corpus based on public domain audio books2015
- Announcing the electromagnetic articulography (day 1) subset of the mngu0 articulatory corpus2011