Kenneth A. Frank

Sensitivity analyses can inform evidence-based policy by quantifying the hypothetical conditions necessary to change an inference. Perhaps the most prevalent index used for sensitivity analyses is Oster’s Coefficient of Proportionality (COP) which expresses how strong selection on unobserved covariates would have to be relative to selection on observed covariates to nullify an estimated effect. But Oster has been critiqued based on its two-stage conceptualization of the COP and its estimation of the COP based on coefficient stability across estimated models. In this article, we reconceptualize the COP as a function of unobserved covariates’ correlations with the focal predictor (e.g., treatment) and with the outcome. Our correlation-based approach addresses the critiques of Oster while preserving the comparison of selection on unobserved covariates to selection on observed covariates. As importantly, our expressions do not depend on analysts’ subjective choices of covariates to include in a baseline model, are adapted to a threshold for inference based on statistical significance, and can be directly calculated from conventionally reported quantities (e.g., estimated effect, standard error) through the Konfound packages in R or Stata or the R-shiny app https://konfound-project.shinyapps.io/konfound-it/ . Thus, for most published studies in the social sciences our correlation-based COP index can be easily applied and intuitively interpreted.

We use evidence from a disruption of clinical documentation routines to propose a novel, predictive mechanism for routine dynamics based on path coherence. Path coherence refers to the continuity of situational attributes from one event to the next along a path, for example, a set of activities conducted by the same person has high actor coherence. Situational attributes include classic descriptors such as who, what, when, where, and why. To be recognized as a path, a minimal level of coherence is required, but path coherence can vary along a path. For example, in a medical clinic, typical paths flow from place to place (e.g., reception, waiting room, exam room) and involve different clinical staff (e.g., receptionist, nurse, physician). Using latent factor network models, we compare clinical documentation routines in five outpatient clinics before and after a technological disruption (an upgrade to the electronic health record system). We show that coherent paths are up to 14 times more likely to persist and up to 40 times more likely to form than less coherent paths. We use these findings to theorize about the role of path coherence in routine dynamics. Path coherence in narrative networks is like homophily in social networks, but with a completely different underlying mechanism. We discuss the implications of our findings for organizational path dependence, resilience, and inertia. Funding: This research was supported by the National Science Foundation [Grants SES-1734237 and BCS-2120530]. This research was also supported in part by the University of Rochester CTSA [Grant UL1 TR002001] from the National Center for Advancing Translational Sciences (NCATS) of the National Institutes of Health (NIH).

The internal validity of a causal inference made based on an observational study is often subject to debate. The potential outcomes framework of causal inference stipulates that causal inference is essentially a missing data problem, and we follow this spirit to define the ideal sample as the combination of the observed data and the missing/counterfactual data for regression models. The robustness of a causal inference can be quantified by the probability of a robust inference for internal validity in regression, i.e., the PIVR, which is the probability of rejecting the null hypothesis again for the ideal sample provided the same null hypothesis has been already rejected for the observed sample. Drawing on the relationship between the PIVR and the mean counterfactual outcomes, we formalize a conceptual framework of quantifying the robustness of a regression-based causal inference based on a joint distribution about the mean counterfactual outcomes, holding the observed sample fixed. Interpretatively, the PIVR is the statistical power of the null hypothesis significance testing that is thought to be built on the ideal sample. We demonstrate the conceptual framework of quantifying the robustness of a regression-based causal inference with an empirical example.

Empirical studies often demonstrate multiple causal mechanisms potentially involving simultaneous or causally related mediators. However, researchers often use simple mediation models to understand the processes because they do not or cannot measure other theoretically relevant mediators. In such cases, another potentially relevant but unobserved mediator potentially confounds the observed mediator, thereby biasing the estimated direct and indirect effects associated with the observed mediator and threatening corresponding inferences. Additionally, researchers may not know the extent to which their measures are reliable, and accordingly, measurement error may bias estimated effects and mislead statistical inferences. Given these threats, we explore how the omission of an unobserved mediator and/or using variables with measurement error biases estimates and affects inferences associated with the observed mediator. Then, building off Frank's impact threshold for a confounding variable (ITCV), we propose a correlation-based sensitivity analysis. Lastly, we provide an R package ConMed to assess the robustness of mediation inferences given the omission of an unobserved, confounding mediator and/or measurement error. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Teacher evaluation policies have been central to policy efforts to enhance teaching quality. At the same time, ambitious mathematics instruction has been emphasized by teacher education programs as well as by the Common Core State Standards. Drawing on observation and survey data from early-career teachers, this study examines how teachers’ perceived pressure of teacher evaluation policies shape their ambitious mathematics instruction. We found that teachers who perceived a strong pressure of teacher evaluation on their instructional practices tended to move further away from enacting ambitious mathematics instruction. Moreover, the negative association between the pressure of teacher evaluation and ambitious instruction was stronger for teachers with a high level of mathematical knowledge for teaching.

This article reports on quantitative findings from a design‐based implementation research (DBIR) project Carbon TIME that focused on teaching carbon cycling at multiple scales, from atomic‐molecular to global. We report results from a large dataset from diverse schools, including 59,654 student assessments of three‐dimensional learning aligned with Next Generation Science Standards collected in classrooms of 133 middle‐ and high‐school teachers over a 4‐year period. We used a subset of these data to investigate factors related to students' learning in this large and diverse set of classrooms. Our findings included: First, Carbon TIME's curriculum and professional development were associated with substantial improvements in overall student achievement; within classrooms students with lower pretest scores had higher learning gains. Second, differences among individual teachers accounted for more of the variance in student learning gains than other measures including school demographics and students' prior knowledge. Third, school demographic factors (percent free and reduced lunch and percent marginalized students of color) had statistically significant but small effects on students' learning. Finally, sustained investment and participation were important; student learning gains improved across years for the project as a whole, and for individual teachers as they gained experience and participated in professional development. These findings contribute to programs of systemic changes to support three‐dimensional science learning.