Allison S. Liu

Gamified educational technologies are often used in education to teach concepts and better motivate students, but questions remain on how to present feedback that supports persistence-related replay learning behaviors. We investigated how providing reward- and error-based feedback related to replay behaviors and problem-solving strategies in From Here to There! (FH2T), an algebra-focused educational technology that utilizes Graspable Math technology and research infrastructure. 1,011 seventh-grade students played nine 30-minute sessions of FH2T. Multilevel logistic regressions examined whether: 1) the number of performance-contingent rewards earned (one to three-star rewards based on solution efficiency) or making at least one error on their first problem-solving attempt predicted students’ choice to replay problems, and 2) replaying was associated with better problem-solving efficiency. Earning two stars was associated with a higher likelihood of replaying compared to one or three stars. Students who replayed problems were more likely to improve their problem-solving efficiency and achieve optimal efficiency on both the replayed problem and the following problem compared to students who did not replay. The option to replay problems without penalty, alongside performance-contingent reward-based feedback, may increase motivation and encourage persistence-related behaviors that are positively associated with learning outcomes. These findings have implications for designing game-like elements for learning and motivation in educational technology and using these contexts as a research infrastructure.

Perceptual learning theory suggests that perceptual grouping in mathematical expressions can direct students' attention toward specific parts of problems, thus impacting their mathematical reasoning. Using in‐lab eye tracking and a sample of 85 undergraduates from a STEM‐focused university, we investigated how higher‐order operator position (HOO; i.e., multiplication/division operators and the presence of superfluous brackets impacted students' time to first fixation to the HOO, response time, and percent of correct responses). Students solved order‐of‐operations problems presented in six ways (3 HOO positions × presence of brackets). We found that HOO position and presence of superfluous brackets had separate and combined impacts on calculating arithmetic expressions. Superfluous brackets most influenced undergraduates' performance when higher‐order operators were located in the center of mathematical expressions. Implications for learning and future directions are discussed about observing eye movements and gaining insights into students' processes when solving arithmetic expressions.

Computer-assisted learning platforms (CALPS) increasingly include gamified elements to improve student outcomes by enhancing their engagement with content. Although evidence exists that gamified programs increase engagement and learning outcomes, there is little causal research on what programmatic mechanisms drive the effect between engagement and learning. In the following paper, we explore this relationship through a method of causal moderation known as fully latent principal stratification. Using data from a large-scale randomized control trial assessing gamified and traditional CALP systems' effects on algebraic knowledge, we estimate the impact of using the gamified CALP on students who engage with one of its key gamification elements---replaying a problem after a suboptimal attempt. The gamified CALP asks students to manipulate algebraic expressions from start to goal states and provides feedback based on the efficiency of these manipulations, allowing students to replay the problems when their efficiency can be improved. We find that the effect of gamification is greater for students with a higher propensity to replay problems. This finding suggests that gamification elements that provide students with opportunities to retry problems are driving the game's efficacy and provide evidence for a scalable mechanism of gamification that can improve students' learning.

Implementation monitoring is an essential component of efficacy studies in education research. Yet traditional methods of fidelity of implementation monitoring can be costly and imprecise. In this paper, we use log-file data from an efficacy study assessing the impact of gamified learning technologies on seventh grade students’ algebraic knowledge, to measure the fidelity of implementations remotely within and across each programs. The COVID-19 pandemic disrupted plans for in-person observations, making reliance on remote monitoring imperative. We conducted an exploratory data analysis and estimated multi-level and latent profile models to explore the use of student level log file data to evaluate the qualities of their implementation contexts. Three metrics emerged as best suited to evaluate the students’ dosage and engagement in their respective programs. The percentage of assignments students started measured students’ dosage. Students’ engagement was best evaluated by two dimensions: the total number of problems they attempted during the study and their average problems completed per assignment. Teachers had the highest intraclass correlations on each fidelity metric, suggesting that in this study the teachers were the most important contextual factor in determining implementation fidelity. Finally, we identified two latent profiles of teachers: high and low fidelity. The analysis presented in this paper functions as both an evaluation of the teacher and student fidelity and as a proof of concept for how to use log-files to remotely monitor and evaluate implementation fidelity in other efficacy studies involving online learning platforms.

As evidence grows supporting the importance of non-cognitive factors in learning, computer-assisted learning platforms increasingly incorporate non-academic interventions to influence student learning and learning related-behaviors. Non-cognitive interventions often attempt to influence students’ mindset, motivation, or metacognitive reflection to impact learning behaviors and outcomes. In the current paper, we analyze data from five experiments, involving seven treatment conditions embedded in mastery-based learning activities hosted on a computer-assisted learning platform focused on middle school mathematics. Each treatment condition embodied a specific non-cognitive theoretical perspective. Over seven school years, 20,472 students participated in the experiments. We estimated the effects of each treatment condition on students’ response time, hint usage, likelihood of mastering knowledge components, learning efficiency, and post-tests performance. Our analyses reveal a mix of both positive and negative treatment effects on student learning behaviors and performance. Few interventions impacted learning as assessed by the post-tests. These findings highlight the difficulty in positively influencing student learning behaviors and outcomes using non-cognitive interventions.

Background: Prior work has shown that middle school students struggle with algebra and that game‐based educational technologies, such as DragonBox and From Here to There!, are effective at improving students' algebraic performance. However, it remains unclear which aspects of algebraic knowledge shift as a result of playing these games and what game design features support algebraic learning.Objective: Using the data from a randomized controlled trial conducted in the 2020–2021 academic year, we (a) examined students' relative performance on assessment items measuring conceptual knowledge, procedural knowledge and procedural flexibility in algebraic equation solving, and (b) identified changes in these aspects of algebraic knowledge after playing DragonBox or From Here to There!.Methods: Eight hundred eighty‐seven seventh‐graders were randomly assigned to playing DragonBox or From Here to There! for nine 30‐min sessions throughout the school year. Students also completed a pretest, midtest and posttest measuring their algebraic knowledge.Results and Conclusion: First, prior to the intervention, students scored the highest on procedural knowledge and lowest on conceptual knowledge. Second, students significantly improved and maintained learning gains on conceptual knowledge throughout the intervention. Their performance on procedural knowledge and procedural flexibility items increased at midtest but decreased at posttest. Third, the pattern of results was consistent for students in the DragonBox and From Here to There! conditions, suggesting that both games may support students' conceptual understanding of algebra. The findings have implications for research and practice on supporting algebraic learning through game‐based technologies during and beyond educational disruptions.

Prior research has shown that game‐based learning tools, such as DragonBox 12+, support algebraic understanding and that students' in‐game progress positively predicts their later performance. Using data from 253 seventh‐graders (12–13 years old) who played DragonBox as a part of technology intervention, we examined (a) the relations between students' progress within DragonBox and their algebraic knowledge and general mathematics achievement, (b) the moderating effects of students' prior performance on these relations and (c) the potential factors associated with students' in‐game progress. Among students with higher prior algebraic knowledge, higher in‐game progress was related to higher algebraic knowledge after the intervention. Higher in‐game progress was also associated with higher end‐of‐year mathematics achievement, and this association was stronger among students with lower prior mathematics achievement. Students' demographic characteristics, prior knowledge and prior achievement did not significantly predict in‐game progress beyond the number of intervention sessions students completed. These findings advance research on how , for whom and in what contexts game‐based interventions, such as DragonBox, support mathematical learning and have implications for practice using game‐based technologies to supplement instruction. Practitioner notes: What is already known about this topic DragonBox 12+ may support students' understanding of algebra but the findings are mixed. Students who solve more problems within math games tend to show higher performance after gameplay. Students' engagement with mathematics is often related to their prior math performance. What this paper adds For students with higher prior algebraic knowledge, solving more problems in DragonBox 12+ is related to higher algebraic performance after gameplay. Students who make more in‐game progress also have higher mathematics achievement, especially for students with lower prior achievement. Students who spend more time playing DragonBox 12+ make more in‐game progress; their demographic, prior knowledge and prior achievement are not related to in‐game progress. Implications for practice and/or policy DragonBox 12+ can be beneficial as a supplement to algebra instruction for students with some understanding of algebra. DragonBox 12+ can engage students with mathematics across achievement levels. Dedicating time and encouraging students to play DragonBox 12+ may help them make more in‐game progress, and in turn, support math learning.

The current study investigated the effectiveness of three distinct educational technologies—two game-based applications (From Here to There and DragonBox 12+) and two modes of online problem sets in ASSISTments (an Immediate Feedback condition and an Active Control condition with no immediate feedback) on Grade 7 students’ algebraic knowledge. More than 3,600 Grade 7 students across nine in-person and one virtual schools within the same district were randomly assigned to one of the four conditions. Students received nine 30-minute intervention sessions from September 2020 to March 2021. Hierarchical linear modeling analyses of the final analytic sample (N = 1,850) showed significantly higher posttest scores for students who used From Here to There and DragonBox 12+ compared to the Active Control condition. No significant difference was found for the Immediate Feedback condition. The findings have implications for understanding how game-based applications can affect algebraic understanding, even within pandemic pressures on learning.