Does Learning Statistics Have to Be So Difficult? Evidence from a Randomized Controlled Trial

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Objectives . Computing education research often relies on empirical methods. Students in this area must use statistical analysis and formalized reporting standards (e.g., APA JARS). These traditions are complex and prior work consistently shows that many learners experience anxiety around them (e.g., data management, methodology, test selection, complex equations, custom programming languages). Most computer science (CS) students receive limited, if any, formal training in statistics beyond an introductory course. In this article, we conduct a randomized controlled trial with students in and out of CS under two conditions. In the first, we use standard statistics traditions along with the programming language R. In the second, we modify these traditions to use more plain language and develop a custom statistics language and API based on a recent theory called “say what you do” that we hypothesize may impact anxiety and correctness. Participants . A total of 293 participants completed the study, including 158 CS students. Non-CS participants were included for comparison in how those in other fields engage with this material. Our sample included different genders and for the first time, to our knowledge, participants with disabilities, a group under-studied regarding statistical anxiety. Study Methods . We conducted a between-subjects experiment that compared a traditional statistical programming approach with an experimental one based on the theory in question. Changes to test names, code structure, and output provided identical answers to R but were redesigned from the lens of CS students that receive little or no training. Participants completed six tasks across three phases of statistical work: selecting tests, writing code, and interpreting output. Measures included anxiety, confidence (self-efficacy), perceived difficulty, uncertainty, and correctness. Quantitative and qualitative responses were collected during each task. Findings . The re-imagined approach reduced anxiety, uncertainty, and perceived difficulty while increasing confidence and performance, especially in coding and interpretation. Disability status and gender significantly influenced statistical anxiety, addressing a gap in research where disability is rarely considered, if at all. Uncertainty emerged as distinct from correctness: participants selected “I am not sure” at a higher rate in the traditional approach, with qualitative feedback suggesting these responses often reflected hesitation rather than misunderstanding. Prior studies classify such responses as incorrect; this work shows that previous performance metrics can conflate knowledge with anxiety-driven disengagement. To our knowledge, this study represents the first formal test of the “say what you do” theory. We found it was largely supported by the evidence, but also that it had limitations that suggest its impact is uneven. Conclusions . Statistics and statistical programming could be altered to be easier for CS students and potentially everyone. In addition to the study, we developed a prototype environment that implements the design principles we evaluated and supports many of the statistical procedures commonly taught in computing education (e.g., ANOVA, common corrections, factor analysis, nonparametric tests). The changes we made reduced statistical anxiety and uncertainty and increased confidence and performance.
Funding Information
  • National Science Foundation (NSF# 2106392 and 2048356)

This publication has 59 references indexed in Scilit: