Replicating Experimental Impact Estimates With Nonexperimental Methods in the Context of Control-Group Noncompliance
Download
A growing literature on within-study comparisons (WSC) examines whether and in what context nonexperimental methods can successfully replicate the results of randomized experiments. WSCs require that the experimental and nonexperimental methods assess the same causal estimand (Cook, Shadish, and Wong, 2008). But experiments that include noncompliance in treatment assignment produce a divergence in the causal estimands measured by standard approaches: the experiment-based estimate of the impact of treatment (the complier average causal effect, CACE) applies only to compliers, while the non-experimental estimate applies to all subjects receiving treatment, including always-takers.
We develop a new replication approach that solves this problem by using nonexperimental methods to produce an estimate that can be compared to the experimental intent-to-treat (ITT) impact estimate rather than the CACE. We demonstrate the applicability of the method in a WSC of the effects of charter schools on student achievement. In our example, some members of the randomized control group crossed over to treatment by enrolling in the charter schools. We show that several nonexperimental methods that incorporate pre-treatment measures of the outcome of interest can successfully replicate experimental ITT impact estimates when control-group noncompliance (crossover) occurs—even when treatment effects differ for compliers and always takers.
More like this from Mathematica
Featured Publication
Evaluation of the Networks for School Improvement Initiative: Interim Summary Report
How do you apply evidence?
Take our quick four-question survey to help us curate evidence and insights that serve you.
Take our survey