Taimoor Arif
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Taimoor Arif has a total of 2 co-authors. Here are the top co-authors by number of shared publications:
- Kevyn Collins-Thompson
School of Information and Dept. of Computer Science and Engineering, University of Michigan, Ann Arbor, MI, USA
1 shared publication - Sumit Asthana
University of Michigan, Ann Abror, MI, USA
1 shared publication
Latest Publications
- Open Access
- 9 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 530-534
Generation and Assessment of Multiple-Choice Questions from Video Transcripts using Large Language Models
We present an empirical study evaluating the quality of multiple-choice questions (MCQs) generated by Large Language Models (LLMs) from a corpus of video transcripts of course lectures in an online data science degree program. With our database of thousands of generated questions, we conducted both human and automated judging of question quality on a representative sample using a broad set of criteria, including well-established Item Writing Flaw (IWF) categories. We found the number of average IWFs per MCQ ranged from 1.6 (rule-based verification) to 2.18 (LLM-based). Among the most frequently identified MCQ flaws were lack of enough context (17%) or answer choices with at least one implausible distractor (57%). Both human and automated assessment identified implausible distractors as one of the most frequent flaw categories. Results from our human annotation study were generally more positive (51--65% good items) compared to our automated assessment study results, which tended toward greater flaw identification (15--25% good items), depending on evaluation method.
- 9 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 530-534
Open AccessGeneration and Assessment of Multiple-Choice Questions from Video Transcripts using Large Language Models
We present an empirical study evaluating the quality of multiple-choice questions (MCQs) generated by Large Language Models (LLMs) from a corpus of video transcripts of course lectures in an online data science degree program. With our database of thousands of generated questions, we conducted both human and automated judging of question quality on a representative sample using a broad set of criteria, including well-established Item Writing Flaw (IWF) categories. We found the number of average IWFs per MCQ ranged from 1.6 (rule-based verification) to 2.18 (LLM-based). Among the most frequently identified MCQ flaws were lack of enough context (17%) or answer choices with at least one implausible distractor (57%). Both human and automated assessment identified implausible distractors as one of the most frequent flaw categories. Results from our human annotation study were generally more positive (51--65% good items) compared to our automated assessment study results, which tended toward greater flaw identification (15--25% good items), depending on evaluation method.