John F. Pane
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
John F. Pane has a total of 18 co-authors. Here are the top co-authors by number of shared publications:
- Adam Chaim Sales
Worcester Polytechnic Institute, Worcester, MA, USA
2 shared publications Daniel F. McCaffrey
Educational Testing Service
2 shared publications- Andrea Phillips
RAND Corporation, Santa Monica, USA
1 shared publication - Anita Israni
University of Texas College of Education University of Texas College of Education RAND Corporation
1 shared publication Annie J. Zhou
RAND Corporation
1 shared publication- Christopher Doss
RAND Corporation, Santa Monica, CA
1 shared publication
Latest Publications
- Open Access
- 25 December 2024
- Published by MDPI AG in Education Sciences
- Vol. 15 (1) , 9
Coding Readiness Assessment: A Measure of Computational Thinking for Preschoolers
Coding and computational thinking (CT) are important skills to include in early childhood education. Measuring these vital skills in preschool-age children can be challenging. We modified the TechCheck-PreK assessment, an unplugged measure of CT for 3-to-5-year-old children, to increase reliability and to add foundational coding concepts. We created the Coding Readiness Assessment (CRA) from a subset of nine TechCheck-PreK items and twelve new items that assess additional CT and coding readiness constructs. In an initial feasibility study of the CRA, teachers observed impulsive responses by children. We mitigated this by implementing a brief delay between the appearance of the question and the timeframe in which children could respond. In a subsequent randomized control trial, the CRA was administered 1637 times by Head Start educators. The assessment took an average of 9.8 min to administer. CRA scores were normally distributed and increased on average as a function of age. Girls scored slightly higher than boys, although the difference was not significant when age and race were taken into account. The CRA showed acceptable levels of reliability in terms of internal consistency (α = 0.78) and test–retest reliability (r = 0.65). Results from a 3PL indicate that the CRA has suitable levels of difficulty and skill level discrimination for 3-to-5-year-olds. The 3PL guessing parameter was 0.28, indicating that steps to reduce impulsive responses were successful. We conclude that the CRA has suitable properties for assessing preschool-age children’s CT skills and coding readiness.
- 25 December 2024
- Published by MDPI AG in Education Sciences
- Vol. 15 (1) , 9
Open AccessCoding Readiness Assessment: A Measure of Computational Thinking for Preschoolers
Coding and computational thinking (CT) are important skills to include in early childhood education. Measuring these vital skills in preschool-age children can be challenging. We modified the TechCheck-PreK assessment, an unplugged measure of CT for 3-to-5-year-old children, to increase reliability and to add foundational coding concepts. We created the Coding Readiness Assessment (CRA) from a subset of nine TechCheck-PreK items and twelve new items that assess additional CT and coding readiness constructs. In an initial feasibility study of the CRA, teachers observed impulsive responses by children. We mitigated this by implementing a brief delay between the appearance of the question and the timeframe in which children could respond. In a subsequent randomized control trial, the CRA was administered 1637 times by Head Start educators. The assessment took an average of 9.8 min to administer. CRA scores were normally distributed and increased on average as a function of age. Girls scored slightly higher than boys, although the difference was not significant when age and race were taken into account. The CRA showed acceptable levels of reliability in terms of internal consistency (α = 0.78) and test–retest reliability (r = 0.65). Results from a 3PL indicate that the CRA has suitable levels of difficulty and skill level discrimination for 3-to-5-year-olds. The 3PL guessing parameter was 0.28, indicating that steps to reduce impulsive responses were successful. We conclude that the CRA has suitable properties for assessing preschool-age children’s CT skills and coding readiness.
- Open Access
- 1 January 2023
- Published by Rand Corporation
Students Using Lexia® Core5® Reading Show Greater Reading Gains Than Matched Comparison Students
This report evaluates the effects of Lexia® Core5® Reading on the reading achievement of students in grades 3–5 during the 2021–2022 academic year. Students who used the program experienced gains in average reading achievement.
- 1 January 2023
- Published by Rand Corporation
Open AccessStudents Using Lexia® Core5® Reading Show Greater Reading Gains Than Matched Comparison Students
This report evaluates the effects of Lexia® Core5® Reading on the reading achievement of students in grades 3–5 during the 2021–2022 academic year. Students who used the program experienced gains in average reading achievement.
- 21 December 2020
- Published by Taylor & Francis in Journal of Research on Educational Effectiveness
- Vol. 14 (1) , 241-269
Student Log-Data from a Randomized Evaluation of Educational Technology: A Causal Case Study
Randomized evaluations of educational technology produce log data as a bi-product: highly granular data on student and teacher usage. These datasets could shed light on causal mechanisms, effect heterogeneity, or optimal use. However, there are methodological challenges: implementation is not randomized and is only defined for the treatment group, and log datasets have a complex structure. This article discusses three approaches to help surmount these issues. One approach uses data from the treatment group to estimate the effect of usage on outcomes in an observational study. Another, causal mediation analysis, estimates the role of usage in driving the overall effect. Finally, principal stratification estimates overall effects for groups of students with the same “potential” usage. We analyze hint data from an evaluation of the Cognitive Tutor Algebra I curriculum using these three approaches, with possibly conflicting results: the observational study and mediation analysis suggest that hints reduce posttest scores, while principal stratification finds that treatment effects may be correlated with higher rates of hint requests. We discuss these mixed conclusions and give broader methodological recommendations.
- 21 December 2020
- Published by Taylor & Francis in Journal of Research on Educational Effectiveness
- Vol. 14 (1) , 241-269
Student Log-Data from a Randomized Evaluation of Educational Technology: A Causal Case Study
Randomized evaluations of educational technology produce log data as a bi-product: highly granular data on student and teacher usage. These datasets could shed light on causal mechanisms, effect heterogeneity, or optimal use. However, there are methodological challenges: implementation is not randomized and is only defined for the treatment group, and log datasets have a complex structure. This article discusses three approaches to help surmount these issues. One approach uses data from the treatment group to estimate the effect of usage on outcomes in an observational study. Another, causal mediation analysis, estimates the role of usage in driving the overall effect. Finally, principal stratification estimates overall effects for groups of students with the same “potential” usage. We analyze hint data from an evaluation of the Cognitive Tutor Algebra I curriculum using these three approaches, with possibly conflicting results: the observational study and mediation analysis suggest that hints reduce posttest scores, while principal stratification finds that treatment effects may be correlated with higher rates of hint requests. We discuss these mixed conclusions and give broader methodological recommendations.
- 1 August 2020
- Published by American Psychological Association (APA) in Psychological Methods
- Vol. 25 (4) , 516-534
Switching cluster membership in cluster randomized control trials: Implications for design and analysis.
Randomized control trials (RCTs) often use clustered designs. where intact clusters (such as classroom. schools, or treatment centers) are randomly assigned to treatment and control conditions. Hierarchical linear models (HLMs) are used almost universally to estimate the effects in such experiments. While study designs that utilize intact clusters have many potential advantages, there is little guidance in the literature on how to respond when cluster switching induces noncompliance with the randomization protocol. In the presence of noncompliance the intent-to-treat (ITT) effect becomes the estimand of interest. When fitting the HLM, these individuals who switch clusters can be assigned to either their as-assigned cluster (the cluster they belonged to at the time of randomization) or their as-treated cluster (the cluster they belonged to at the time the outcome was collected). We show analytically and via simulation. that using the as-treated cluster in HLM will bias the estimate of the ITT effect and using the as-assigned cluster will bias the standard error estimates when heterogeneity among clusters is because of heterogeneity in treatment effects. We show that using linear regression with two-way cluster adjusted standard errors can yield unbiased ITT estimates and consistent standard errors regardless of the source of the random effects. We recommend this method replace HLM as the method of choice for testing intervention effects with cluster-randomized trials with noncompliance and cluster switching. Translational Abstract Randomized control trials (RCTs) often use clustered designs, where intact clusters (such as classrooms. schools, clinics, or treatment centers) are randomly assigned to treatment and control conditions. Hierarchical linear models (HLMs) are used almost universally to estimate the impact of a program or intervention in such experiments. While study designs that utilize intact clusters have many potential advantages, there is little guidance for researchers or practitioners on how to respond when some individuals move between clusters and "break" the randomization protocol. In this case, individuals who switch clusters can be assigned to either their as-assigned cluster (the cluster they belonged to at the time of randomization) or their as-treated cluster (the cluster they belonged to at the time the outcome was collected). We show that using the as-treated cluster in HLM will bias treatment effect estimates and using the as-assigned cluster will bias standard error estimates when there are differences in treatment effects across clusters. We propose an alternative strategy that yields unbiased estimates and appropriate standard errors for cluster-randomized trials with cluster switching.
- 1 August 2020
- Published by American Psychological Association (APA) in Psychological Methods
- Vol. 25 (4) , 516-534
Switching cluster membership in cluster randomized control trials: Implications for design and analysis.
Randomized control trials (RCTs) often use clustered designs. where intact clusters (such as classroom. schools, or treatment centers) are randomly assigned to treatment and control conditions. Hierarchical linear models (HLMs) are used almost universally to estimate the effects in such experiments. While study designs that utilize intact clusters have many potential advantages, there is little guidance in the literature on how to respond when cluster switching induces noncompliance with the randomization protocol. In the presence of noncompliance the intent-to-treat (ITT) effect becomes the estimand of interest. When fitting the HLM, these individuals who switch clusters can be assigned to either their as-assigned cluster (the cluster they belonged to at the time of randomization) or their as-treated cluster (the cluster they belonged to at the time the outcome was collected). We show analytically and via simulation. that using the as-treated cluster in HLM will bias the estimate of the ITT effect and using the as-assigned cluster will bias the standard error estimates when heterogeneity among clusters is because of heterogeneity in treatment effects. We show that using linear regression with two-way cluster adjusted standard errors can yield unbiased ITT estimates and consistent standard errors regardless of the source of the random effects. We recommend this method replace HLM as the method of choice for testing intervention effects with cluster-randomized trials with noncompliance and cluster switching. Translational Abstract Randomized control trials (RCTs) often use clustered designs, where intact clusters (such as classrooms. schools, clinics, or treatment centers) are randomly assigned to treatment and control conditions. Hierarchical linear models (HLMs) are used almost universally to estimate the impact of a program or intervention in such experiments. While study designs that utilize intact clusters have many potential advantages, there is little guidance for researchers or practitioners on how to respond when some individuals move between clusters and "break" the randomization protocol. In this case, individuals who switch clusters can be assigned to either their as-assigned cluster (the cluster they belonged to at the time of randomization) or their as-treated cluster (the cluster they belonged to at the time the outcome was collected). We show that using the as-treated cluster in HLM will bias treatment effect estimates and using the as-assigned cluster will bias standard error estimates when there are differences in treatment effects across clusters. We propose an alternative strategy that yields unbiased estimates and appropriate standard errors for cluster-randomized trials with cluster switching.
- 27 February 2020
- Published by Springer Nature in Educational Technology Research and Development
- Vol. 68 (3) , 1409-1437
Implementing an adaptive intelligent tutoring system as an instructional supplement
- 27 February 2020
- Published by Springer Nature in Educational Technology Research and Development
- Vol. 68 (3) , 1409-1437
Implementing an adaptive intelligent tutoring system as an instructional supplement
- 13 May 2019
- Published by American Educational Research Association (AERA) in Educational Researcher
- Vol. 48 (4) , 217-228
Translating Standardized Effects of Education Programs Into More Interpretable Metrics
Evaluators report effects of education initiatives as standardized effect sizes, a scale that has merits but obscures interpretation of the effects’ practical importance. Consequently, educators and policymakers seek more readily interpretable translations of evaluation results. One popular metric is the number of years of learning necessary to induce the effect. We compare years of learning to three other translation options: benchmarking against other effect sizes, converting to percentile growth, and estimating the probability of scoring above a proficiency threshold. After enumerating the desirable properties of translations, we examine each option’s strengths and weaknesses. We conclude that years of learning performs worst, and percentile gains performs best, making it our recommended choice for more interpretable translations of standardized effects.
- 13 May 2019
- Published by American Educational Research Association (AERA) in Educational Researcher
- Vol. 48 (4) , 217-228
Translating Standardized Effects of Education Programs Into More Interpretable Metrics
Evaluators report effects of education initiatives as standardized effect sizes, a scale that has merits but obscures interpretation of the effects’ practical importance. Consequently, educators and policymakers seek more readily interpretable translations of evaluation results. One popular metric is the number of years of learning necessary to induce the effect. We compare years of learning to three other translation options: benchmarking against other effect sizes, converting to percentile growth, and estimating the probability of scoring above a proficiency threshold. After enumerating the desirable properties of translations, we examine each option’s strengths and weaknesses. We conclude that years of learning performs worst, and percentile gains performs best, making it our recommended choice for more interpretable translations of standardized effects.
- Open Access
Mastery Learning in Practice: A (Mostly) Descriptive Analysis of Log Data from the Cognitive Tutor Algebra I Effectiveness Trial
Mastery learning, the notion that students learn best if they move on from studying a topic only after having demonstrated mastery, sits at the foundation of the theory of intelligent tutoring. This paper is an exploration of how mastery learning plays out in practice, based on log data from a large randomized effectiveness trial of the Cognitive Tutor Algebra I (CTAI) curriculum. We find that students frequently progressed from CTAI sections they were working on without demonstrating mastery and worked units out of order. Moreover, these behaviors were substantially more common in the second year of the study, in which the CTAI effect was significantly larger. We explore the various ways students departed from the official CTAI curriculum, focusing on heterogeneity between years, states, schools, and students. The paper concludes with an observational study of the effect on post-test scores of teachers reassigning students out of their current sections before they mastered the requisite skills, finding that reassignment appears to lowers posttest scores--a finding that is fairly resilient to confounding from omitted covariates--but that the effect varies substantially between classrooms.
Open AccessMastery Learning in Practice: A (Mostly) Descriptive Analysis of Log Data from the Cognitive Tutor Algebra I Effectiveness Trial
Mastery learning, the notion that students learn best if they move on from studying a topic only after having demonstrated mastery, sits at the foundation of the theory of intelligent tutoring. This paper is an exploration of how mastery learning plays out in practice, based on log data from a large randomized effectiveness trial of the Cognitive Tutor Algebra I (CTAI) curriculum. We find that students frequently progressed from CTAI sections they were working on without demonstrating mastery and worked units out of order. Moreover, these behaviors were substantially more common in the second year of the study, in which the CTAI effect was significantly larger. We explore the various ways students departed from the official CTAI curriculum, focusing on heterogeneity between years, states, schools, and students. The paper concludes with an observational study of the effect on post-test scores of teachers reassigning students out of their current sections before they mastered the requisite skills, finding that reassignment appears to lowers posttest scores--a finding that is fairly resilient to confounding from omitted covariates--but that the effect varies substantially between classrooms.
- 1 October 2008
- Published by Taylor & Francis in Journal of Education for Students Placed at Risk (JESPAR)
- Vol. 13 (2-3) , 168-211
Effects of Student Displacement in Louisiana During the First Academic Year After the Hurricanes of 2005
The combined impact of Hurricanes Katrina and Rita in 2005 left the education system in the Gulf Coast region of the United States with tremendous challenges to rebuild infrastructure, reestablish services, and accommodate the movements of students that occurred as result of the storms. This article focuses on Louisiana's public school students, of whom 1/4 were displaced as a result of the storms. It explores the experiences of students and the effects of student movements on student achievement and on the state's public education system during the first academic year following the hurricanes. Results suggest that, overall, the education system handled the disruptions of the disaster relatively well. However, more detailed examinations show that some displaced students had problems such as nonenrollment or poor attendance, mental health or behavioral problems, and academic setbacks. Negative achievement effects, which were small overall, were most pronounced among students who remained displaced for the duration of the academic year, and appeared to be mitigated by students' tendency to enroll in schools with higher student performance than their original schools. Additional results show that the negative effects were associated with both the number of schools attended and the amount of school time lost. However, these analyses do not include the displaced students who did not reenroll in a Louisiana public school. These students constitute more than a 1/4 of displaced students and tended to be those who are most at risk for poor academic outcomes.
- 1 October 2008
- Published by Taylor & Francis in Journal of Education for Students Placed at Risk (JESPAR)
- Vol. 13 (2-3) , 168-211
Effects of Student Displacement in Louisiana During the First Academic Year After the Hurricanes of 2005
The combined impact of Hurricanes Katrina and Rita in 2005 left the education system in the Gulf Coast region of the United States with tremendous challenges to rebuild infrastructure, reestablish services, and accommodate the movements of students that occurred as result of the storms. This article focuses on Louisiana's public school students, of whom 1/4 were displaced as a result of the storms. It explores the experiences of students and the effects of student movements on student achievement and on the state's public education system during the first academic year following the hurricanes. Results suggest that, overall, the education system handled the disruptions of the disaster relatively well. However, more detailed examinations show that some displaced students had problems such as nonenrollment or poor attendance, mental health or behavioral problems, and academic setbacks. Negative achievement effects, which were small overall, were most pronounced among students who remained displaced for the duration of the academic year, and appeared to be mitigated by students' tendency to enroll in schools with higher student performance than their original schools. Additional results show that the negative effects were associated with both the number of schools attended and the amount of school time lost. However, these analyses do not include the displaced students who did not reenroll in a Louisiana public school. These students constitute more than a 1/4 of displaced students and tended to be those who are most at risk for poor academic outcomes.
- 6 January 2003
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 213-220
The design view: a design oriented, high-level visual programming environment
- 6 January 2003
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 213-220
The design view: a design oriented, high-level visual programming environment