Uncovering Bias in ASR Systems: Evaluating Wav2vec2 and Whisper for Dutch speakers
Authors:
- Iskaj JanssenIJIskaj Janssen
- 25 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 146-151
Abstract
It is crucial that ASR systems can handle the wide range of variations in speech of speakers from different demographic groups, with different speaking styles, and of speakers with (dis)abilities. A potential quality-of-service harm arises when ASR systems do not perform equally well for everyone. ASR systems may exhibit bias against certain types of speech, such as non-native accents, different age groups and gender. In this study, we evaluate two widely-used neural network-based architectures: Wav2vec2 and Whisper on potential biases for Dutch speakers. We used the Dutch speech corpus JASMIN as a test set containing read and conversational speech in a human-machine interaction setting. The results reveal a significant bias against non-natives, children and elderly and some regional dialects. The ASR systems generally perform slightly better for women than for men.Keywords
This publication has 6 references indexed in Scilit:
- Hey ASR System! Why Aren’t You More Inclusive?2022
- Investigating the Impact of Gender Representation in ASR Training Data: a Case Study on Librispeech2021
- Racial disparities in automated speech recognitionProceedings of the National Academy of Sciences, 2020
- Automated Speech Recognition Systems and Older Adults: A Literature Review and SynthesisProceedings of the Human Factors and Ergonomics Society Annual Meeting, 2019
- Gender Representation in French Broadcast Corpora and Its Impact on ASR Performance2019
- Gender and Dialect Bias in YouTube's Automatic Captions2017