Robust neural tracking of linguistic speech representations using a convolutional neural network
- 1 August 2023
- journal article
- research article
- Published by IOP Publishing in Journal of Neural Engineering
- Vol. 20 (4) , 046040
Abstract
Objective. When listening to continuous speech, populations of neurons in the brain track different features of the signal. Neural tracking can be measured by relating the electroencephalography (EEG) and the speech signal. Recent studies have shown a significant contribution of linguistic features over acoustic neural tracking using linear models. However, linear models cannot model the nonlinear dynamics of the brain. To overcome this, we use a convolutional neural network (CNN) that relates EEG to linguistic features using phoneme or word onsets as a control and has the capacity to model non-linear relations. Approach. We integrate phoneme- and word-based linguistic features (phoneme surprisal, cohort entropy (CE), word surprisal (WS) and word frequency (WF)) in our nonlinear CNN model and investigate if they carry additional information on top of lexical features (phoneme and word onsets). We then compare the performance of our nonlinear CNN with that of a linear encoder and a linearized CNN. Main results. For the non-linear CNN, we found a significant contribution of CE over phoneme onsets and of WS and WF over word onsets. Moreover, the non-linear CNN outperformed the linear baselines. Significance. Measuring coding of linguistic features in the brain is important for auditory neuroscience research and applications that involve objectively measuring speech understanding. With linear models, this is measurable, but the effects are very small. The proposed non-linear CNN model yields larger differences between linguistic and lexical models and, therefore, could show effects that would otherwise be unmeasurable and may, in the future, lead to improved within-subject measures and shorter recordings.Keywords
All Related Versions
Funding Information
- KU Leuven (C24/18/099)
- Fonds Wetenschappelijk Onderzoek (1290821N)
This publication has 26 references indexed in Scilit:
- The Multivariate Temporal Response Function (mTRF) Toolbox: A MATLAB Toolbox for Relating Neural Signals to Continuous StimuliFrontiers in Human Neuroscience, 2016
- Low-Frequency Cortical Entrainment to Speech Reflects Phoneme-Level ProcessingCurrent Biology, 2015
- Attentional Selection in a Cocktail Party Environment Can Be Decoded from Single-Trial EEGCerebral Cortex, 2014
- Auditory Brainstem Response to Complex Sounds Predicts Self-Reported Speech-in-Noise PerformanceJournal of Speech, Language, and Hearing Research, 2013
- Emergence of neural encoding of auditory objects while listening to competing speakersProceedings of the National Academy of Sciences, 2012
- SUBTLEX-NL: A new measure for Dutch word frequency based on film subtitlesBehavior Research Methods, 2010
- Developing a reading tutor: Design and evaluation of dedicated speech recognition and synthesis modulesSpeech Communication, 2009
- APEX 3: a multi-purpose test platform for auditory psychophysical experimentsJournal of Neuroscience Methods, 2008
- Estimating Audiometric Thresholds Using Auditory Steady-State ResponsesJournal of the American Academy of Audiology, 2005
- Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentencesIEEE Transactions on Acoustics, Speech, and Signal Processing, 1980