Prerna Rao
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Prerna Rao has a total of 9 co-authors. Here are the top co-authors by number of shared publications:
- Murtaza Ali
Human-Centered Design and Engineering, University of Washington, Seattle, Washington, USA
2 shared publications - Alix Medler
Human-centered Design and Engineering, University of Washington, USA
1 shared publication - Benjamin Xie
University of Denver, Denver, Colorado, USA
1 shared publication - Mengqi Shi
University of Washington, Seattle, Washington, USA
1 shared publication - Raveena Dhegaskar
Human-centered Design and Engineering, University of Washington, USA
1 shared publication - Sayamindu Dasgupta
University of Washington, Seattle, Washington, USA
1 shared publication
Latest Publications
- Open Access
- 12 August 2024
- Published by Association for Computing Machinery (ACM)
- p. 452-468
Using Benchmarking Infrastructure to Evaluate LLM Performance on CS Concept Inventories: Challenges, Opportunities, and Critiques
BACKGROUND AND CONTEXT. The pace of advancement of large language models (LLMs) motivates the use of existing infrastructure to automate the evaluation of LLM performance on computing education tasks. Concept inventories are well suited for evaluation because of their careful design and prior validity evidence. OBJECTIVES. Our research explores the feasibility of using an automated benchmarking framework to evaluate computer science (CS) concept inventories. We explore three primary objectives: evaluation of LLM performance on the SCS1 and BDSI concept inventories; informal expert panel review of items which had variations between LLM and expected student performance; and description of challenges with using benchmarking infrastructure as a methodological innovation. METHOD. We used the Holistic Evaluation of Language Models (HELM) framework to evaluate the SCS1 and BDSI against 10 LLMS with zero-shot and few-shot in-context learning: GPT (3.5, 4.0), Claude (1.3, 2.0, 2.1), Llama (7B, 13B, 70B), Mistral v0.1 7B, and Mixtral 8x7B. We used psychometric data from prior studies to measure knowledge levels for each LLM run. We then conducted an informal expert review to qualitatively explore how question design, CS content knowledge, and LLM design may explain differences between LLM and expected student performances. FINDINGS. Our quantitative analysis found that most LLM response patterns reflected a below average introductory computing student with the SCS1 and did not fit the psychometric 2PL model for the BDSI. Our qualitative analysis identified that LLMs performed well on code infill questions, but poorly on nested conditionals, runtime analysis, and longer questions. We also identified several methodological challenges related to item security, translation, the structure when using HELM. IMPLICATIONS. We consider the feasibility of using automated benchmarking as a methodology to support more reproducible, replicable, and rigorous investigations to understand the intersection of LLM capabilities, computing concepts, and assessment design. We also consider connections between psychometric approaches and LLM evaluations to inform the design of computing assessments that are more resilient to LLM advancements.
- 12 August 2024
- Published by Association for Computing Machinery (ACM)
- p. 452-468
Open AccessUsing Benchmarking Infrastructure to Evaluate LLM Performance on CS Concept Inventories: Challenges, Opportunities, and Critiques
BACKGROUND AND CONTEXT. The pace of advancement of large language models (LLMs) motivates the use of existing infrastructure to automate the evaluation of LLM performance on computing education tasks. Concept inventories are well suited for evaluation because of their careful design and prior validity evidence. OBJECTIVES. Our research explores the feasibility of using an automated benchmarking framework to evaluate computer science (CS) concept inventories. We explore three primary objectives: evaluation of LLM performance on the SCS1 and BDSI concept inventories; informal expert panel review of items which had variations between LLM and expected student performance; and description of challenges with using benchmarking infrastructure as a methodological innovation. METHOD. We used the Holistic Evaluation of Language Models (HELM) framework to evaluate the SCS1 and BDSI against 10 LLMS with zero-shot and few-shot in-context learning: GPT (3.5, 4.0), Claude (1.3, 2.0, 2.1), Llama (7B, 13B, 70B), Mistral v0.1 7B, and Mixtral 8x7B. We used psychometric data from prior studies to measure knowledge levels for each LLM run. We then conducted an informal expert review to qualitatively explore how question design, CS content knowledge, and LLM design may explain differences between LLM and expected student performances. FINDINGS. Our quantitative analysis found that most LLM response patterns reflected a below average introductory computing student with the SCS1 and did not fit the psychometric 2PL model for the BDSI. Our qualitative analysis identified that LLMs performed well on code infill questions, but poorly on nested conditionals, runtime analysis, and longer questions. We also identified several methodological challenges related to item security, translation, the structure when using HELM. IMPLICATIONS. We consider the feasibility of using automated benchmarking as a methodology to support more reproducible, replicable, and rigorous investigations to understand the intersection of LLM capabilities, computing concepts, and assessment design. We also consider connections between psychometric approaches and LLM evaluations to inform the design of computing assessments that are more resilient to LLM advancements.
- Open Access
- 7 August 2023
- Published by Association for Computing Machinery (ACM)
- p. 397-415
Taking Stock of Concept Inventories in Computing Education: A Systematic Literature Review
Background and context. Concept inventories (CIs) are a widely used tool in STEM education that can help instructors identify specific misconceptions students hold about key concepts. Over the past several years, much research has been published contributing to CIs in computer science education. Objectives. This research aims to investigate the state of CIs in computer science education through a systematic literature review built around two research questions: 1) What do we know about CIs for computer science today, and 2) Have researchers been able to effectively tackle the 4 challenges specific to computer science CIs outlined by Taylor et al. in their previous 2014 literature review on this topic? Method. We conducted a systematic literature review focusing on CIs in computer science education. We included research from any country, provided it was published in English between 2004 and 2022. We searched with key terms across four databases (Compendex, Inspec, Education Source, and ERIC) and collated a final list of 65 studies for analysis from 175 initial results. Findings. From these 65 papers, we identified 33 total CIs in computer science, 12 of which are currently validated. Of the 65 studies, the majority used established techniques for choosing concepts, identifying misconceptions, and/or developing questions, but a few (n = 4) reported on successful alternative techniques. We found that although no papers explicitly articulated an objective to tackle the aforementioned challenges, they are a recurring theme in the literature and have been addressed in various ways. Implications. At the time of the previous literature review, there were few computer science CIs. Our findings are promising, suggesting that researchers have found effective ways to tackle Taylor et al.’s challenges and that the number of CIs across computer science is gradually increasing. Future work is warranted regarding non-traditional techniques researchers proposed for developing CIs.
- 7 August 2023
- Published by Association for Computing Machinery (ACM)
- p. 397-415
Open AccessTaking Stock of Concept Inventories in Computing Education: A Systematic Literature Review
Background and context. Concept inventories (CIs) are a widely used tool in STEM education that can help instructors identify specific misconceptions students hold about key concepts. Over the past several years, much research has been published contributing to CIs in computer science education. Objectives. This research aims to investigate the state of CIs in computer science education through a systematic literature review built around two research questions: 1) What do we know about CIs for computer science today, and 2) Have researchers been able to effectively tackle the 4 challenges specific to computer science CIs outlined by Taylor et al. in their previous 2014 literature review on this topic? Method. We conducted a systematic literature review focusing on CIs in computer science education. We included research from any country, provided it was published in English between 2004 and 2022. We searched with key terms across four databases (Compendex, Inspec, Education Source, and ERIC) and collated a final list of 65 studies for analysis from 175 initial results. Findings. From these 65 papers, we identified 33 total CIs in computer science, 12 of which are currently validated. Of the 65 studies, the majority used established techniques for choosing concepts, identifying misconceptions, and/or developing questions, but a few (n = 4) reported on successful alternative techniques. We found that although no papers explicitly articulated an objective to tackle the aforementioned challenges, they are a recurring theme in the literature and have been addressed in various ways. Implications. At the time of the previous literature review, there were few computer science CIs. Our findings are promising, suggesting that researchers have found effective ways to tackle Taylor et al.’s challenges and that the number of CIs across computer science is gradually increasing. Future work is warranted regarding non-traditional techniques researchers proposed for developing CIs.