Rebecca Maynard
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Rebecca Maynard has a total of 35 co-authors. Here are the top co-authors by number of shared publications:
- Jessaca Spybrook
Western Michigan University, Kalamazoo, MI, USA
3 shared publications Nianbo Dong
3 shared publications- Dustin Anderson
EVALCORP, Irvine, USA
2 shared publications Irma Perez-Johnson
University of Pennsylvania
2 shared publicationsMatthew Stagner
Urban Institute
2 shared publications- Nianbo Dong
University of North Carolina at Chapel Hill
2 shared publications
Latest Publications
- Open Access
- 3 February 2024
- Published by SAGE Publications in Evaluation Review
- Vol. 48 (3) , 515-543
Improving the Usefulness and Use of Meta-Analysis to Inform Policy and Practice
This chapter begins with an overview of recent developments that have encouraged and facilitated greater use of research syntheses, including Meta-Analysis, to guide public policy and practice in education, workforce development, and social services. It discusses the role of Meta-Analysis for improving knowledge of the effectiveness of programs, policies, and practices and the applicability and generalizability of that knowledge to conditions other than those represented by the study samples and settings. The chapter concludes with recommendations for improving the potential of Meta-Analysis to accelerate knowledge development through changing how we design, conduct, and report findings of individual studies to maximize their usefulness in Meta-Analysis as well as how we produce and report Meta-Analysis findings. The paper includes references to resources supporting the recommendations.
- 3 February 2024
- Published by SAGE Publications in Evaluation Review
- Vol. 48 (3) , 515-543
Open AccessImproving the Usefulness and Use of Meta-Analysis to Inform Policy and Practice
This chapter begins with an overview of recent developments that have encouraged and facilitated greater use of research syntheses, including Meta-Analysis, to guide public policy and practice in education, workforce development, and social services. It discusses the role of Meta-Analysis for improving knowledge of the effectiveness of programs, policies, and practices and the applicability and generalizability of that knowledge to conditions other than those represented by the study samples and settings. The chapter concludes with recommendations for improving the potential of Meta-Analysis to accelerate knowledge development through changing how we design, conduct, and report findings of individual studies to maximize their usefulness in Meta-Analysis as well as how we produce and report Meta-Analysis findings. The paper includes references to resources supporting the recommendations.
- 19 December 2022
- Published by Taylor & Francis in Journal of Research on Educational Effectiveness
- Vol. 16 (4) , 681-706
Experimental Design and Statistical Power for Cluster Randomized Cost-Effectiveness Trials
Cluster randomized trials (CRTs) are commonly used to evaluate educational interventions, particularly their effectiveness. Recently there has been greater emphasis on using these trials to explore cost-effectiveness. However, methods for establishing the power of cluster randomized cost-effectiveness trials (CRCETs) are limited. This study develops power computation formulas and statistical software to help researchers plan two- and three-level CRCETs. We illustrate the application of our formulas and software for the designs of CRCETs and discuss the influence of sample size, nesting effects, covariates, and the covariance between cost and effectiveness measures on the statistical power of cost-effectiveness estimates.
- 19 December 2022
- Published by Taylor & Francis in Journal of Research on Educational Effectiveness
- Vol. 16 (4) , 681-706
Experimental Design and Statistical Power for Cluster Randomized Cost-Effectiveness Trials
Cluster randomized trials (CRTs) are commonly used to evaluate educational interventions, particularly their effectiveness. Recently there has been greater emphasis on using these trials to explore cost-effectiveness. However, methods for establishing the power of cluster randomized cost-effectiveness trials (CRCETs) are limited. This study develops power computation formulas and statistical software to help researchers plan two- and three-level CRCETs. We illustrate the application of our formulas and software for the designs of CRCETs and discuss the influence of sample size, nesting effects, covariates, and the covariance between cost and effectiveness measures on the statistical power of cost-effectiveness estimates.
- 12 August 2021
- Published by Springer Nature in Prevention Science
- Vol. 23 (5) , 764-773
Study Registration for the Field of Prevention Science: Considering Options and Paths Forward
The practice of prospectively registering the details of intervention studies in a public database or registry is gaining momentum across disciplines as a strategy for increasing the transparency, credibility, and accessibility of study findings. In this article, we consider five registries that may be relevant for registration of intervention studies in the field of prevention science: ClinicalTrials.gov, the American Economic Association Registry of Randomized Controlled Trials (AEA RCT Registry), the Open Science Framework Preregistration (OSF Preregistration), the Registry for International Development Impact Evaluations (RIDIE), and the Registry of Efficacy and Effectiveness Studies (REES). We examine the five registries in terms of substantive focus, study designs, and contents of registry entries. We consider two paths forward for prospective registration of intervention studies in the field of prevention science: Path A: register all studies in ClinicalTrials.gov and Path B: allow individual researchers to select the registry with the “best fit.” Lastly, we consider how the field might begin to establish norms around registration.
- 12 August 2021
- Published by Springer Nature in Prevention Science
- Vol. 23 (5) , 764-773
Study Registration for the Field of Prevention Science: Considering Options and Paths Forward
The practice of prospectively registering the details of intervention studies in a public database or registry is gaining momentum across disciplines as a strategy for increasing the transparency, credibility, and accessibility of study findings. In this article, we consider five registries that may be relevant for registration of intervention studies in the field of prevention science: ClinicalTrials.gov, the American Economic Association Registry of Randomized Controlled Trials (AEA RCT Registry), the Open Science Framework Preregistration (OSF Preregistration), the Registry for International Development Impact Evaluations (RIDIE), and the Registry of Efficacy and Effectiveness Studies (REES). We examine the five registries in terms of substantive focus, study designs, and contents of registry entries. We consider two paths forward for prospective registration of intervention studies in the field of prevention science: Path A: register all studies in ClinicalTrials.gov and Path B: allow individual researchers to select the registry with the “best fit.” Lastly, we consider how the field might begin to establish norms around registration.
- 28 May 2020
- Published by SAGE Publications in Evaluation Review
- Vol. 46 (5) , 469-516
Using Iterative Experimentation to Accelerate Program Improvement: A Case Example
Background: This article offers a case example of how experimental evaluation methods can be coupled with principles of design-based implementation research (DBIR), improvement science (IS), and rapid-cycle evaluation (RCE) methods to provide relatively quick, low-cost, credible assessments of strategies designed to improve programs, policies, or practices. Objectives: This article demonstrates the feasibility and benefits of blending DBIR, IS, and RCE practices with embedded randomized controlled trials (RCTs) to improve the pace and efficiency of program improvement. Research design: This article describes a two-cycle experimental test of staff-designed strategies for improving a workforce development program. Youth enrolled in Year Up’s Professional Training Corps (PTC) programs were randomly assigned to “improvement strategies” designed to boost academic success and persistence through the 6-month learning and development (L&D) phase of the program, when participants spend most of their program-related time in courses offered by partner colleges. Subjects: The study sample includes 317 youth from three PTC program sites. Measures: The primary outcome measures are completion of the program’s L&D phase and continued college enrollment beyond the L&D phase. Results: The improvement strategies designed and tested during the study increased program retention through L&D by nearly 10 percentage points and increased college persistence following L&D by 13 percentage points. Conclusion: Blending DBIR, IS, and RCE principles with a multi-cycle RCT generated highly credible estimates of the efficacy of the tested improvement strategies within a relatively short period of time (18 months) at modest cost and with reportedly low burden for program staff.
- 28 May 2020
- Published by SAGE Publications in Evaluation Review
- Vol. 46 (5) , 469-516
Using Iterative Experimentation to Accelerate Program Improvement: A Case Example
Background: This article offers a case example of how experimental evaluation methods can be coupled with principles of design-based implementation research (DBIR), improvement science (IS), and rapid-cycle evaluation (RCE) methods to provide relatively quick, low-cost, credible assessments of strategies designed to improve programs, policies, or practices. Objectives: This article demonstrates the feasibility and benefits of blending DBIR, IS, and RCE practices with embedded randomized controlled trials (RCTs) to improve the pace and efficiency of program improvement. Research design: This article describes a two-cycle experimental test of staff-designed strategies for improving a workforce development program. Youth enrolled in Year Up’s Professional Training Corps (PTC) programs were randomly assigned to “improvement strategies” designed to boost academic success and persistence through the 6-month learning and development (L&D) phase of the program, when participants spend most of their program-related time in courses offered by partner colleges. Subjects: The study sample includes 317 youth from three PTC program sites. Measures: The primary outcome measures are completion of the program’s L&D phase and continued college enrollment beyond the L&D phase. Results: The improvement strategies designed and tested during the study increased program retention through L&D by nearly 10 percentage points and increased college persistence following L&D by 13 percentage points. Conclusion: Blending DBIR, IS, and RCE principles with a multi-cycle RCT generated highly credible estimates of the efficacy of the tested improvement strategies within a relatively short period of time (18 months) at modest cost and with reportedly low burden for program staff.
- 27 March 2020
- Published by American Educational Research Association (AERA) in Journal of Educational and Behavioral Statistics
- Vol. 45 (6) , 690-718
Power Analysis for Two-Level Multisite Randomized Cost-Effectiveness Trials
Cost-effectiveness analysis is a widely used educational evaluation tool. The randomized controlled trials that aim to evaluate the cost-effectiveness of the treatment are commonly referred to as randomized cost-effectiveness trials (RCETs). This study provides methods of power analysis for two-level multisite RCETs. Power computations take account of sample sizes, the effect size, covariates effects, nesting effects for both cost and effectiveness measures, the ratio of the total variance of the cost measure to the total variance of effectiveness measure, and correlations between cost and effectiveness measures at each level. Illustrative examples that show how power is influenced by the sample sizes, nesting effects, covariate effects, and correlations between cost and effectiveness measures are presented. We also demonstrate how the calculations can be applied in the design phase of two-level multisite RCETs using the software PowerUp!-CEA (Version 1.0).
- 27 March 2020
- Published by American Educational Research Association (AERA) in Journal of Educational and Behavioral Statistics
- Vol. 45 (6) , 690-718
Power Analysis for Two-Level Multisite Randomized Cost-Effectiveness Trials
Cost-effectiveness analysis is a widely used educational evaluation tool. The randomized controlled trials that aim to evaluate the cost-effectiveness of the treatment are commonly referred to as randomized cost-effectiveness trials (RCETs). This study provides methods of power analysis for two-level multisite RCETs. Power computations take account of sample sizes, the effect size, covariates effects, nesting effects for both cost and effectiveness measures, the ratio of the total variance of the cost measure to the total variance of effectiveness measure, and correlations between cost and effectiveness measures at each level. Illustrative examples that show how power is influenced by the sample sizes, nesting effects, covariate effects, and correlations between cost and effectiveness measures are presented. We also demonstrate how the calculations can be applied in the design phase of two-level multisite RCETs using the software PowerUp!-CEA (Version 1.0).
- 8 November 2018
- Published by American Educational Research Association (AERA) in Educational Researcher
- Vol. 48 (1) , 45-50
REES: A Registry of Efficacy and Effectiveness Studies in Education
Calls for improving transparency across the social science are increasing. One strategy that is gaining momentum in the quest to increase transparency is the practice of preregistering a study and analysis plan prior to conducting a study. In this article, we examine the potential of preregistration as a strategy for increasing transparency in education studies. We review existing registries within the social sciences and provide a rationale for why we need a registry specifically geared towards education studies. Finally, we introduce the Registry of Efficacy and Effectiveness Studies (REES), developed by the Society for Research on Educational Effectiveness (SREE). The goal of REES is to increase transparency for studies seeking to draw causal conclusions within the education research and evaluation community.
- 8 November 2018
- Published by American Educational Research Association (AERA) in Educational Researcher
- Vol. 48 (1) , 45-50
REES: A Registry of Efficacy and Effectiveness Studies in Education
Calls for improving transparency across the social science are increasing. One strategy that is gaining momentum in the quest to increase transparency is the practice of preregistering a study and analysis plan prior to conducting a study. In this article, we examine the potential of preregistration as a strategy for increasing transparency in education studies. We review existing registries within the social sciences and provide a rationale for why we need a registry specifically geared towards education studies. Finally, we introduce the Registry of Efficacy and Effectiveness Studies (REES), developed by the Society for Research on Educational Effectiveness (SREE). The goal of REES is to increase transparency for studies seeking to draw causal conclusions within the education research and evaluation community.
- 9 August 2018
- Published by Taylor & Francis
- p. 285-337
The Costs of Adolescent Childbearing
The measured economic consequences of young teen childbearing fall into several broad categories. Out-of-pocket costs borne by parents for children's health care result from a combination of the fertility effects of early childbearing and the health consequences for children of being born to adolescent mothers. The per-child consequences of early childbearing for out-of-pocket health care costs for children are measured directly as part of the Kids Having Kids research. The costs of young teen childbearing to society pertain only to those consequences that reflect real changes in the resources available for consumption by the population at large. A portion of the higher health care costs is due to the larger family sizes, a portion is due to the higher fraction of care that is subsidized, and still another portion is due to the worse health conditions of children of young teen parents.
- 9 August 2018
- Published by Taylor & Francis
- p. 285-337
The Costs of Adolescent Childbearing
The measured economic consequences of young teen childbearing fall into several broad categories. Out-of-pocket costs borne by parents for children's health care result from a combination of the fertility effects of early childbearing and the health consequences for children of being born to adolescent mothers. The per-child consequences of early childbearing for out-of-pocket health care costs for children are measured directly as part of the Kids Having Kids research. The costs of young teen childbearing to society pertain only to those consequences that reflect real changes in the resources available for consumption by the population at large. A portion of the higher health care costs is due to the larger family sizes, a portion is due to the higher fraction of care that is subsidized, and still another portion is due to the worse health conditions of children of young teen parents.
- 18 June 2018
- Published by SAGE Publications in The Annals of the American Academy of Political and Social Science
- Vol. 678 (1) , 134-144
The Role of Federal Agencies in Creating and Administering Evidence-Based Policies
The success of federal agencies in creating and using evidence-based policies hinges on (1) their commitment to include routine use of evidence—including research and program evaluations—in program design and funding decisions and (2) their capacity to adapt their operating practices accordingly. The recent push toward using evidence more deliberately in government meant that federal agencies needed to quickly improve the accessibility of existing evidence. They also had to foster internal capacity to fairly judge its quality and applicability; build capacity and support for routinely using evidence within program and policy offices to support policy development and monitoring; and create a consensus within agencies around sensible ways to categorize, rate, and apply evidence. Common evidence standards, open access to evidence review platforms, and mandates for embedding rigorous evaluations into funded programs are among the most influential tools agencies have used in this new era of evidence-based policymaking.
- 18 June 2018
- Published by SAGE Publications in The Annals of the American Academy of Political and Social Science
- Vol. 678 (1) , 134-144
The Role of Federal Agencies in Creating and Administering Evidence-Based Policies
The success of federal agencies in creating and using evidence-based policies hinges on (1) their commitment to include routine use of evidence—including research and program evaluations—in program design and funding decisions and (2) their capacity to adapt their operating practices accordingly. The recent push toward using evidence more deliberately in government meant that federal agencies needed to quickly improve the accessibility of existing evidence. They also had to foster internal capacity to fairly judge its quality and applicability; build capacity and support for routinely using evidence within program and policy offices to support policy development and monitoring; and create a consensus within agencies around sensible ways to categorize, rate, and apply evidence. Common evidence standards, open access to evidence review platforms, and mandates for embedding rigorous evaluations into funded programs are among the most influential tools agencies have used in this new era of evidence-based policymaking.
- 6 December 2016
- Published by Wiley in New Directions for Evaluation
- Vol. 2016 (152) , 109-135
Program and Policy Evaluations in Practice: Highlights from the Federal Perspective
This paper examines the intersection of evaluation methods and usefulness of research for policy and practice from the vantage point of federal agencies that commission a large share of domestic program evaluations with the goal of improving the ability of the government to invest its scarce dollars wisely. Toward this end, the paper revisits the prominent issues discussed in prior chapters through a lens focused on understanding conditions under which particular methodological strategies are and are not helpful in advancing the job of policy analysts, policymakers, and program administrators. The aim is to provide useful guidance to evaluators in how to interact productively with the evaluation funders and intended end‐users regarding how they might make their products most useful.
- 6 December 2016
- Published by Wiley in New Directions for Evaluation
- Vol. 2016 (152) , 109-135
Program and Policy Evaluations in Practice: Highlights from the Federal Perspective
This paper examines the intersection of evaluation methods and usefulness of research for policy and practice from the vantage point of federal agencies that commission a large share of domestic program evaluations with the goal of improving the ability of the government to invest its scarce dollars wisely. Toward this end, the paper revisits the prominent issues discussed in prior chapters through a lens focused on understanding conditions under which particular methodological strategies are and are not helpful in advancing the job of policy analysts, policymakers, and program administrators. The aim is to provide useful guidance to evaluators in how to interact productively with the evaluation funders and intended end‐users regarding how they might make their products most useful.
- 7 July 2015
- Published by SAGE Publications in American Journal of Evaluation
- Vol. 36 (4) , 558-569
Unlocking the Potential of the “What Works” Approach to Policymaking and Practice
Despite bipartisan support in Washington, DC, which dates back to the mid-1990s, the “what works” approach has yet to gain broad support among policymakers and practitioners. One way to build such support is to increase the usefulness of program impact evaluations for these groups. We describe three ways to make impact evaluations more useful to policy and practice: emphasize learning from all studies over sorting out winners and losers; collect better information on the conditions that shape an intervention's success or failure; and learn about the features of programs and policies that influence their effectiveness. We argue that measurement of the treatment contrast that exists between the intervention and comparison condition(s) is important for each of these changes. Measurement and analysis of the treatment contrast will increase cost and policymakers and practitioners already see evaluations as expensive. Therefore we offer suggestions for reducing costs in other areas of data collection.
- 7 July 2015
- Published by SAGE Publications in American Journal of Evaluation
- Vol. 36 (4) , 558-569
Unlocking the Potential of the “What Works” Approach to Policymaking and Practice
Despite bipartisan support in Washington, DC, which dates back to the mid-1990s, the “what works” approach has yet to gain broad support among policymakers and practitioners. One way to build such support is to increase the usefulness of program impact evaluations for these groups. We describe three ways to make impact evaluations more useful to policy and practice: emphasize learning from all studies over sorting out winners and losers; collect better information on the conditions that shape an intervention's success or failure; and learn about the features of programs and policies that influence their effectiveness. We argue that measurement of the treatment contrast that exists between the intervention and comparison condition(s) is important for each of these changes. Measurement and analysis of the treatment contrast will increase cost and policymakers and practitioners already see evaluations as expensive. Therefore we offer suggestions for reducing costs in other areas of data collection.