Hynek Hermansky
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Hynek Hermansky has a total of 93 co-authors. Here are the top co-authors by number of shared publications:
Samuel Thomas
Human Language Technology Center of Excellence, Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD USA
6 shared publicationsSriram Ganapathy
Center for Language and Speech Processing, Department of Electrical and Computer Engineering, The Johns Hopkins University, USA
4 shared publicationsSri Harish Mallidi
3 shared publicationsAren Jansen
Human Language Technology Center of Excellence, Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD USA
2 shared publicationsBernd T. Meyer
2 shared publications- Feipeng Li
School of Information and Communications Engineering and the National Engineering Research Center for Offshore Oil and Gas Exploration, Xian Jiaotong University, Xian, Shaanxi, China
2 shared publications
Latest Publications
- Open Access
Self-paced silent speech brain-computer interface for device control
Brain-computer interfaces can potentially restore autonomy to people with paralysis, including the ability to control their own environment via smart devices in the home. BCI applications for controlling smart devices to date have required visual displays or auditory cues, limiting the user’s ability to autonomously and privately issue commands. In this study, a clinical trial participant with amyotrophic lateral sclerosis (ALS) used a chronically implanted electrocorticographic (ECoG) BCI to control smart devices with self-paced silent speech commands. Across 18 experimental sessions, silently mimed speech commands were detected in real time and decoded with a median accuracy of 97.1% (chance: 7.14%). These results demonstrate that silently attempted speech can be reliably decoded without exogenous timing cues, supporting the feasibility of reliable autonomous smart device control with an implantable BCI.
Open AccessSelf-paced silent speech brain-computer interface for device control
Brain-computer interfaces can potentially restore autonomy to people with paralysis, including the ability to control their own environment via smart devices in the home. BCI applications for controlling smart devices to date have required visual displays or auditory cues, limiting the user’s ability to autonomously and privately issue commands. In this study, a clinical trial participant with amyotrophic lateral sclerosis (ALS) used a chronically implanted electrocorticographic (ECoG) BCI to control smart devices with self-paced silent speech commands. Across 18 experimental sessions, silently mimed speech commands were detected in real time and decoded with a median accuracy of 97.1% (chance: 7.14%). These results demonstrate that silently attempted speech can be reliably decoded without exogenous timing cues, supporting the feasibility of reliable autonomous smart device control with an implantable BCI.
- Open Access
- 1 August 2024
- Published by IOP Publishing in Journal of Neural Engineering
- Vol. 21 (4) , 046056
Iterative alignment discovery of speech-associated neural activity
Objective. Brain–computer interfaces (BCIs) have the potential to preserve or restore speech in patients with neurological disorders that weaken the muscles involved in speech production. However, successful training of low-latency speech synthesis and recognition models requires alignment of neural activity with intended phonetic or acoustic output with high temporal precision. This is particularly challenging in patients who cannot produce audible speech, as ground truth with which to pinpoint neural activity synchronized with speech is not available. Approach. In this study, we present a new iterative algorithm for neural voice activity detection (nVAD) called iterative alignment discovery dynamic time warping (IAD-DTW) that integrates DTW into the loss function of a deep neural network (DNN). The algorithm is designed to discover the alignment between a patient’s electrocorticographic (ECoG) neural responses and their attempts to speak during collection of data for training BCI decoders for speech synthesis and recognition. Main results. To demonstrate the effectiveness of the algorithm, we tested its accuracy in predicting the onset and duration of acoustic signals produced by able-bodied patients with intact speech undergoing short-term diagnostic ECoG recordings for epilepsy surgery. We simulated a lack of ground truth by randomly perturbing the temporal correspondence between neural activity and an initial single estimate for all speech onsets and durations. We examined the model’s ability to overcome these perturbations to estimate ground truth. IAD-DTW showed no notable degradation (Significance. IAD-DTW is computationally inexpensive and can be easily integrated into existing DNN-based nVAD approaches, as it pertains only to the final loss computation. This approach makes it possible to train speech BCI algorithms using ECoG data from patients who are unable to produce audible speech, including those with Locked-In Syndrome.
- 1 August 2024
- Published by IOP Publishing in Journal of Neural Engineering
- Vol. 21 (4) , 046056
Open AccessIterative alignment discovery of speech-associated neural activity
Objective. Brain–computer interfaces (BCIs) have the potential to preserve or restore speech in patients with neurological disorders that weaken the muscles involved in speech production. However, successful training of low-latency speech synthesis and recognition models requires alignment of neural activity with intended phonetic or acoustic output with high temporal precision. This is particularly challenging in patients who cannot produce audible speech, as ground truth with which to pinpoint neural activity synchronized with speech is not available. Approach. In this study, we present a new iterative algorithm for neural voice activity detection (nVAD) called iterative alignment discovery dynamic time warping (IAD-DTW) that integrates DTW into the loss function of a deep neural network (DNN). The algorithm is designed to discover the alignment between a patient’s electrocorticographic (ECoG) neural responses and their attempts to speak during collection of data for training BCI decoders for speech synthesis and recognition. Main results. To demonstrate the effectiveness of the algorithm, we tested its accuracy in predicting the onset and duration of acoustic signals produced by able-bodied patients with intact speech undergoing short-term diagnostic ECoG recordings for epilepsy surgery. We simulated a lack of ground truth by randomly perturbing the temporal correspondence between neural activity and an initial single estimate for all speech onsets and durations. We examined the model’s ability to overcome these perturbations to estimate ground truth. IAD-DTW showed no notable degradation (Significance. IAD-DTW is computationally inexpensive and can be easily integrated into existing DNN-based nVAD approaches, as it pertains only to the final loss computation. This approach makes it possible to train speech BCI algorithms using ECoG data from patients who are unable to produce audible speech, including those with Locked-In Syndrome.
- Open Access
Complex Frequency Domain Linear Prediction: A Tool to Compute Modulation Spectrum of Speech
Conventional Frequency Domain Linear Prediction (FDLP) technique models the squared Hilbert envelope of speech with varied degrees of approximation which can be sampled at the required frame rate and used as features for Automatic Speech Recognition (ASR). Although previously the complex cepstrum of the conventional FDLP model has been used as compact frame-wise speech features, it has lacked interpretability in the context of the Hilbert envelope. In this paper, we propose a modification of the conventional FDLP model that allows easy interpretability of the complex cepstrum as temporal modulations in an all-pole model approximation of the power of the speech signal. Additionally, our "complex" FDLP yields significant speed-ups in comparison to conventional FDLP for the same degree of approximation.
Open AccessComplex Frequency Domain Linear Prediction: A Tool to Compute Modulation Spectrum of Speech
Conventional Frequency Domain Linear Prediction (FDLP) technique models the squared Hilbert envelope of speech with varied degrees of approximation which can be sampled at the required frame rate and used as features for Automatic Speech Recognition (ASR). Although previously the complex cepstrum of the conventional FDLP model has been used as compact frame-wise speech features, it has lacked interpretability in the context of the Hilbert envelope. In this paper, we propose a modification of the conventional FDLP model that allows easy interpretability of the complex cepstrum as temporal modulations in an all-pole model approximation of the power of the speech signal. Additionally, our "complex" FDLP yields significant speed-ups in comparison to conventional FDLP for the same degree of approximation.
- Open Access
Radically Old Way of Computing Spectra: Applications in End-to-End ASR
We propose a technique to compute spectrograms using Frequency Domain Linear Prediction (FDLP) that uses all-pole models to fit the squared Hilbert envelope of speech in different frequency sub-bands. The spectrogram of a complete speech utterance is computed by overlap-add of contiguous all-pole model responses. A long context window of 1.5 seconds allows us to capture the low frequency temporal modulations of speech in the spectrogram. For an end-to-end automatic speech recognition task, the FDLP spectrogram performs on par with the standard mel spectrogram features for clean read speech training and test data. For more realistic speech data with train-test domain mismatches or reverberations, FDLP spectrogram shows up to 25% and 22% relative WER improvements over mel spectrogram respectively.
Open AccessRadically Old Way of Computing Spectra: Applications in End-to-End ASR
We propose a technique to compute spectrograms using Frequency Domain Linear Prediction (FDLP) that uses all-pole models to fit the squared Hilbert envelope of speech in different frequency sub-bands. The spectrogram of a complete speech utterance is computed by overlap-add of contiguous all-pole model responses. A long context window of 1.5 seconds allows us to capture the low frequency temporal modulations of speech in the spectrogram. For an end-to-end automatic speech recognition task, the FDLP spectrogram performs on par with the standard mel spectrogram features for clean read speech training and test data. For more realistic speech data with train-test domain mismatches or reverberations, FDLP spectrogram shows up to 25% and 22% relative WER improvements over mel spectrogram respectively.
- 13 December 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 28 (23299290) , 646-655
Multi-Stream End-to-End Speech Recognition
Attention-based methods and Connectionist Temporal Classification (CTC) network have been promising research directions for end-to-end (E2E) Automatic Speech Recognition (ASR). The joint CTC/Attention model has achieved great success by utilizing both architectures during multi-task training and joint decoding. In this work, we present a multi-stream framework based on joint CTC/Attention E2E ASR with parallel streams represented by separate encoders aiming to capture diverse information. On top of the regular attention networks, the Hierarchical Attention Network (HAN) is introduced to steer the decoder toward the most informative encoders. A separate CTC network is assigned to each stream to force monotonic alignments. Two representative framework have been proposed and discussed, which are Multi-Encoder Multi-Resolution (MEM-Res) framework and Multi-Encoder Multi-Array (MEM-Array) framework, respectively. In MEM-Res framework, two heterogeneous encoders with different architectures, temporal resolutions and separate CTC networks work in parallel to extract complimentary information from same acoustics. Experiments are conducted on Wall Street Journal (WSJ) and CHiME-4, resulting in relative Word Error Rate (WER) reduction of 18.0-32.1% and the best WER of 3.6% in the WSJ eval92 test set. The MEM-Array framework aims at improving the far-field ASR robustness using multiple microphone arrays which are activated by separate encoders. Compared with the best single-array results, the proposed framework has achieved relative WER reduction of 3.7% and 9.7% in AMI and DIRHA multi-array corpora, respectively, which also outperforms conventional fusion strategies.
- 13 December 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 28 (23299290) , 646-655
Multi-Stream End-to-End Speech Recognition
Attention-based methods and Connectionist Temporal Classification (CTC) network have been promising research directions for end-to-end (E2E) Automatic Speech Recognition (ASR). The joint CTC/Attention model has achieved great success by utilizing both architectures during multi-task training and joint decoding. In this work, we present a multi-stream framework based on joint CTC/Attention E2E ASR with parallel streams represented by separate encoders aiming to capture diverse information. On top of the regular attention networks, the Hierarchical Attention Network (HAN) is introduced to steer the decoder toward the most informative encoders. A separate CTC network is assigned to each stream to force monotonic alignments. Two representative framework have been proposed and discussed, which are Multi-Encoder Multi-Resolution (MEM-Res) framework and Multi-Encoder Multi-Array (MEM-Array) framework, respectively. In MEM-Res framework, two heterogeneous encoders with different architectures, temporal resolutions and separate CTC networks work in parallel to extract complimentary information from same acoustics. Experiments are conducted on Wall Street Journal (WSJ) and CHiME-4, resulting in relative Word Error Rate (WER) reduction of 18.0-32.1% and the best WER of 3.6% in the WSJ eval92 test set. The MEM-Array framework aims at improving the far-field ASR robustness using multiple microphone arrays which are activated by separate encoders. Compared with the best single-array results, the proposed framework has achieved relative WER reduction of 3.7% and 9.7% in AMI and DIRHA multi-array corpora, respectively, which also outperforms conventional fusion strategies.
- Open Access
- 1 December 2019
- Published by Acoustical Society of America (ASA) in The Journal of the Acoustical Society of America
- Vol. 146 (6) , EL459-EL463
General properties of auditory spectro-temporal receptive fields
The analyses of more than 2000 spectro-temporal receptive fields (STRFs) obtained from the ferret primary auditory cortex revealed their dominant encoding properties along time and frequency domains. Results showed that the peak responses of the STRFs were roughly aligned along the time axis, and enhanced low modulation components of the signal around 3 Hz. On the contrary, the peaks of the STRF population along the frequency axis varied and were distributed widely. Further analyses revealed that there were some general properties along the frequency axis around the best frequency of each STRF. The STRFs enhanced modulation frequencies around 0.25 cycles/octave. These findings are consistent with techniques that have been reported in the literature to result in improvements in speech recognition performance, highlighting the significant potential of biological insights into the design of better machines.
- 1 December 2019
- Published by Acoustical Society of America (ASA) in The Journal of the Acoustical Society of America
- Vol. 146 (6) , EL459-EL463
Open AccessGeneral properties of auditory spectro-temporal receptive fields
The analyses of more than 2000 spectro-temporal receptive fields (STRFs) obtained from the ferret primary auditory cortex revealed their dominant encoding properties along time and frequency domains. Results showed that the peak responses of the STRFs were roughly aligned along the time axis, and enhanced low modulation components of the signal around 3 Hz. On the contrary, the peaks of the STRF population along the frequency axis varied and were distributed widely. Further analyses revealed that there were some general properties along the frequency axis around the best frequency of each STRF. The STRFs enhanced modulation frequencies around 0.25 cycles/octave. These findings are consistent with techniques that have been reported in the literature to result in improvements in speech recognition performance, highlighting the significant potential of biological insights into the design of better machines.
- Open Access
Exploring Methods for the Automatic Detection of Errors in Manual Transcription
Quality of data plays an important role in most deep learning tasks. In the speech community, transcription of speech recording is indispensable. Since the transcription is usually generated artificially, automatically finding errors in manual transcriptions not only saves time and labors but benefits the performance of tasks that need the training process. Inspired by the success of hybrid automatic speech recognition using both language model and acoustic model, two approaches of automatic error detection in the transcriptions have been explored in this work. Previous study using a biased language model approach, relying on a strong transcription-dependent language model, has been reviewed. In this work, we propose a novel acoustic model based approach, focusing on the phonetic sequence of speech. Both methods have been evaluated on a completely real dataset, which was originally transcribed with errors and strictly corrected manually afterwards.
Open AccessExploring Methods for the Automatic Detection of Errors in Manual Transcription
Quality of data plays an important role in most deep learning tasks. In the speech community, transcription of speech recording is indispensable. Since the transcription is usually generated artificially, automatically finding errors in manual transcriptions not only saves time and labors but benefits the performance of tasks that need the training process. Inspired by the success of hybrid automatic speech recognition using both language model and acoustic model, two approaches of automatic error detection in the transcriptions have been explored in this work. Previous study using a biased language model approach, relying on a strong transcription-dependent language model, has been reviewed. In this work, we propose a novel acoustic model based approach, focusing on the phonetic sequence of speech. Both methods have been evaluated on a completely real dataset, which was originally transcribed with errors and strictly corrected manually afterwards.
- 1 January 2019
- Published by Elsevier in Speech Communication
- Vol. 106, 112-117
Coding and decoding of messages in human speech communication: Implications for machine recognition of speech
- 1 January 2019
- Published by Elsevier in Speech Communication
- Vol. 106, 112-117
Coding and decoding of messages in human speech communication: Implications for machine recognition of speech
- 1 January 2019
- Published by Elsevier in Speech Communication
- Vol. 106, 44-56
DNN-based performance measures for predicting error rates in automatic speech recognition and optimizing hearing aid parameters
- 1 January 2019
- Published by Elsevier in Speech Communication
- Vol. 106, 44-56
DNN-based performance measures for predicting error rates in automatic speech recognition and optimizing hearing aid parameters