Modeling Word Importance in Conversational Transcripts: Toward improved live captioning for Deaf and hard of hearing viewers
Open Access
- 30 April 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- Vol. 2023, 79-83
Abstract
Despite the recent improvements in automatic speech recognition (ASR) systems, their accuracy is imperfect in live conversational settings. Classifying the importance of each word in a caption transcription can enable evaluation metrics that best reflect Deaf and Hard of Hearing (DHH) readers’ judgment of the caption quality. Prior work has proposed using word embeddings, e.g., word2vec or BERT embeddings, to model word importance in conversational transcripts. Recent work also disseminated a human-annotated word importance dataset. We conducted a word-token level analysis on this dataset and explored Part-of-Speech (POS) distribution. We then augmented the dataset with POS tags and reduced the class imbalance by generating 5% additional text using masking. Finally, we investigated how various supervised models learn the importance of words. The best performing model trained on our augmented dataset performed better than prior models. Our findings can inform the design of a metric for measuring live caption quality from DHH users’ perspectives.Keywords
Funding Information
- National Science Foundation (DGE-2125362)
This publication has 9 references indexed in Scilit:
- Caption-occlusion severity judgments across live-television genres from deaf and hard-of-hearing viewers2021
- Context-Aware Document Term Weighting for Ad-Hoc Search2020
- YAKE! Keyword extraction from single documents using multiple local featuresInformation Sciences, 2020
- Evaluating the Benefit of Highlighting Key Words in Captions for People who are Deaf or Hard of Hearing2019
- Predicting the Understandability of Imperfect English Captions for People Who Are Deaf or Hard of HearingACM Transactions on Accessible Computing, 2019
- Word Error Rate Estimation for Speech Recognition: e-WER2018
- Improving the Estimation of Word Importance for News Multi-Document Summarization2014
- TF–IDF2011
- NLTK2002