Education Policy Research

Reports

• "Impacts of Five Expeditionary Learning Middle Schools on Academic Achievement." Ira Nichols-Barrer and Joshua Haimson, July 2013. In the first rigorous study of the impacts of Expeditionary Learning (EL) model schools, Mathematica found that EL middle school students perform better in reading and math than their counterparts in other public schools. Fact sheet.
• "The Effectiveness of Secondary Math Teachers from Teach For America and the Teaching Fellows Programs." Melissa A. Clark, Hanley S. Chiang, Tim Silva, Sheena McConnell, Kathy Sonnenfeld, Anastasia Erbe, and Michael Puma, September 2013. The first large-scale, random assignment study of the effects of secondary school math teachers from Teach For America and the Teaching Fellows programs found they were as effective as, and in some cases more effective than, teachers receiving traditional certification. The study was sponsored by the U.S. Department of Education's Institute of Education Sciences. Read the brief.
• "Improving Post-High School Outcomes for Transition-Age Students with Disabilities: An Evidence Review." R. Brian Cobb, Stephen Lipscomb, Jennifer Wolgemuth, Theresa Schulte, Abigail Veliquette, Morgen Alwell, Keriu Batchelder, Robert Bernard, Paul Hernandez, Helen Holmquist-Johnson, Rebecca Orsi, Laura Sample McMeeking, Jun Wang, and Andrea Welnberg, August 2013. This report uses evidence-based standards developed by the What Works Clearinghouse to review the research literature on programs that help students with disabilities make transitions after high school. Community-based work programs had mixed effects on employment and potentially positive effects on postsecondary education. Functional life skills development programs had potentially positive effects on independent living although the extent of evidence was small. Executive Summary
• "Classroom Observations from Phase 2 of the Pennsylvania Teacher Evaluation Pilot: Assessing Internal Consistency, Score Variation, and Relationships with Value Added." Elias Walsh and Stephen Lipscomb, May 2013. This report presents findings from Phase 2 of a three-year teacher evaluation pilot conducted by the Pennsylvania Department of Education. Principals evaluated the teaching practices of teachers using The Framework for Teaching, a rubric that includes 22 components grouped into four broad teaching practice domains: (1) planning and preparation, (2) classroom environment, (3) instruction, and (4) professional responsibilities. Although principals did not typically use all 22 components, the report’s findings suggest the fairness of overall scores might not be compromised substantially by principals using different sets of components. Also, across nearly all components, teachers with higher scores on the rubric tended to make larger contributions to student achievement than did teachers with lower scores, as measured by value added. The report’s findings suggest that the rubric measures aspects of teachers’ practices related to growth in student achievement on standardized assessments.
• "Impacts of Five Expeditionary Learning Middle Schools on Academic Achievement." Ira Nichols-Barrer and Joshua Haimson, July 2013.
  
• "KIPP Middle Schools: Impacts on Achievement and Other Outcomes." Christina Clark Tuttle, Brian Gill, Philip Gleason, Virginia Knechtel, Ira Nichols-Barrer, and Alexandra Resch, February 2013. This report shows that Knowledge Is Power Program (KIPP) middle schools have significant and substantial positive impacts on student achievement in four core academic subjects: reading, math, science, and social studies. One of the report’s analyses confirms the positive impacts using a rigorous randomized experimental analysis that relies on the schools’ admissions lotteries to identify comparison students, thereby accounting for students’ prior achievement, as well as factors such as student and parent motivation. The latest findings from Mathematica’s multiyear study of KIPP middle schools, the report is the most rigorous large-scale evaluation of KIPP charter schools to date, covering 43 KIPP middle schools in 13 states and the District of Columbia. Student outcomes examined included state test results in reading and math, test scores in science and social studies, results on a nationally normed assessment that includes measures of higher-order thinking, and behaviors reported by students and parents. Executive summary. Fact sheet.
• "Value-Added Estimates for Phase 1 of the Pennsylvania Teacher and Principal Evaluation Pilot." Stephen Lipscomb, Hanley Chiang, and Brian Gill, April 2012. This report describes the development of value-added models for estimating the contributions of Pennsylvania teachers and principals toward the achievement growth of their students. Estimates were obtained during the first phase of a multiyear pilot to develop new evaluation systems for teachers and principals. The report also examines whether teachers with higher classroom observation scores on specific professional practices among those who participated in the first phase tended to have greater impacts on student achievement, as measured by value-added models. Executive Summary. Technical Report.
• "Replicating Experimental Impact Estimates Using a Regression Discontinuity Approach." Philip M. Gleason, Alexandra M. Resch, and Jillian A. Berk, April 2012. This report attempts to replicate the estimated impacts from experimental studies of two different education interventions using a regression discontinuity design. In each case, the estimated impact of the intervention based on the regression discontinuity design was not significantly different from the experimental impact estimate, though the magnitude of the differences between the point estimates of impacts from the two designs sometimes was nontrivial.
• "Impacts of Title I Supplemental Educational Services on Student Achievement." John Deke, Lisa Dragoset, Karen Bogen, and Brian Gill, May 2012. As part of No Child Left Behind, parents of low-income students in low-performing schools are offered Supplemental Educational Services (SES) for their children. These academic supports, such as extra tutoring or group sessions, take place outside the regular school day. This report for the Institute of Education Sciences examine potential achievement benefits. In the six study districts located in Connecticut, Florida, and Ohio, the program was directed to the lowest-achieving students due to oversubscription. However, not all students who were offered access to the program participated. The study found no evidence of impacts from offering SES to students near the cutoff of acceptance into the program. Furthermore, there were no impacts from participating in SES on student achievement in reading or math. Providers offered an average of 21 hours of SES per student for the school year, either one-on-one or in group sessions conducted by local teachers. No observed provider characteristics and practices, including intensity of services, were significantly associated with stronger impacts. The six districts were not nationally representative. Executive summary.

Journal Articles

• "Borrowing Constraints, College Enrollment, and Delayed Entry." Matthew T. Johnson. Journal of Labor Economics, October 2013 (subscription required). This article specifies a dynamic model of education, borrowing, and work decisions of high school graduates to ascertain how increasing the amount students are permitted to borrow through government-sponsored loan programs would affect educational attainment.
• "Statistical Power for School-Based RCTs with Binary Outcomes." Peter Z. Schochet. Journal of Research on Educational Effectiveness, June 2013 (subscription required). This article develops a new approach for calculating appropriate sample sizes for school-based randomized controlled trials (RCTs) with binary outcomes using logit models with and without baseline covariates. The theoretical analysis develops sample size formulas for clustered designs in which random assignment is at the school or teacher level using generalized estimating equation methods. The key finding is that sample sizes of 40 to 60 schools that are typically included in clustered RCTs for student test score or behavioral scale outcomes will often be insufficient for binary outcomes.
• "Funding Special Education by Total District Enrollment: Advantages, Disadvantages, and Policy Considerations." Elizabeth Dhuey and Stephen Lipscomb. Education Finance and Policy, summer 2013 (subscription required). This policy brief aims to help policymakers, educators, and researchers better understand census funding, a special education finance model used by several states and the federal government. Under this model, aid levels are based primarily on total district enrollment and a fixed amount per student. Census funding is viewed as a cost-containment approach, but it has raised concerns about funding equity. The brief examines the key advantages and disadvantages of the model and discusses options for easing funding equity concerns. It also describes other ways in which states and districts may be able to contain special education costs while maintaining quality programs.
• "What Are Error Rates for Classifying Teacher and School Performance Using Value-Added Models?" Peter Z. Schochet and Hanley S. Chiang. Journal of Educational and Behavioral Statistics, April 2013 (subscription required). This article addresses likely error rates for measuring teacher and school performance in the upper elementary grades using value-added models applied to student test score gain data. Using formulas based on ordinary least squares and empirical Bayes estimators, error rates for comparing a teacher’s performance to the average are likely to be about 25 percent with three years of data and 35 percent with one year of data. Corresponding error rates for overall false positive and negative errors are 10 percent and 20 percent, respectively. The results suggest that policymakers must carefully consider likely system error rates when using value-added estimates to make high-stakes decisions regarding educators.
• "Combination Classes and Educational Achievement." Jaime L. Thomas. Economics of Education Review, December 2012 (subscription required). This article examines the relationship between combination class membership in 1st grade and 1st-grade test scores, finding that 1st graders are not harmed by being in a combination class or by their schools offering combination classes. As long as other stakeholders—such as parents, teachers, and students in other grades—are not made worse off, these findings suggest that offering combination classes may be a viable cost-saving option for school administrators.
• "Effectiveness of Four Supplemental Reading Comprehension Interventions." Susanne James-Burdumy, John Deke, Russell Gersten, Julieta Lugo-Gil, Rebecca Newman-Gonchar, Joseph Dimino, Kelly Haymond, and Albert Yung-Hsu Liu. Journal of Research on Educational Effectiveness, October 2012 (subscription required). This article presents evidence from a large-scale randomized controlled trial of the effects of four supplemental reading comprehension curricula (Project CRISS, ReadAbout, Read for Real, and Reading for Knowledge). The impact analyses in the study's first year revealed a statistically significant negative impact of Reading for Knowledge on students' reading comprehension scores and no other significant impacts. The impact of ReadAbout was positive and significant in the study's second year among teachers with one year of experience using the intervention.
• "Random Assignment Within Schools: Lessons Learned from the Teach For America Experiment." Steven Glazerman. Education Finance and Policy, April 2012 (subscription required). This article discusses the trade-offs associated with study designs that involve random assignment of students within schools and describes the experience from one such study of Teach For America (TFA). The article concludes that within-school random assignment studies such as the TFA evaluation are challenging but may also be feasible and generate useful evidence.

Issue Briefs

• "Supporting Policy and Program Decisions: Recommendations for Conducting High Quality Systematic Evidence Reviews." Susan Zief and Roberto Agodini, October 2012. Systematic reviews are a useful tool for decision makers because they identify relevant studies about the effectiveness of a policy or program of interest, assess the quality of the evidence from the relevant studies, and summarize the valid findings. Mathematica's Center for Improving Research Evidence recently released an issue brief offering recommendations for conducting high quality systematic reviews. The recommendations aim to increase the number of such reviews in order to provide decision makers with a greater number of useful evidence summaries that can inform policy and program decisions.
• "Findings from a Randomized Experiment of Playworks: Selected Results from Cohort 1." Martha Bleeker, Susanne James-Burdumy, Nicholas Beyler, Allison Hedley Dodd, Rebecca A. London, Lisa Westrich, Katie Stokes-Guinan, and Sebastian Castrechini, April 2012. Findings based on the first cohort of schools included several significant, positive impacts. Playworks had a positive impact on teachers' perceptions of students' safety and feeling more included during recess. Teachers in Playworks schools also reported less bullying and exclusionary behavior during recess, and found transitions from recess to classroom learning were less difficult than teachers in control schools found. Teachers in Playworks schools reported significantly better student behavior at recess and readiness for class than teachers in control schools and were also more likely to report that their students enjoyed adult-organized recess activities. Students in Playworks schools reported better behavior and attention in class after sports, games, and play than students in control schools. Overall, most teachers, students, and principals reported positive perceptions of the Playworks program.
• "Do Low-Income Students Have Equal Access to the Highest-Performing Teachers?" Steven Glazerman and Jeffrey Max, April 2011. Most research on equal educational opportunity has focused on inputs like teacher experience and degrees. This brief estimated teachers’ value added (contribution to student achievement growth) and measured access to highest-performing teachers in high- and low-poverty schools. Across 10 selected districts in seven states students in the highest-poverty schools had unequal access, on average, to the district’s highest-performing middle school teachers. The pattern for elementary school was less clear. The degree of equal access varied by district. Technical Appendix.
• "Infusing Academics into Career and Technical Education." Trends in Education Research, Issue Brief #3. Joshua Haimson, James R. Stone, III, and Donna Pearson, December 2008. Integrating academic learning into career and technical education (CTE) classes can be challenging for educators and curriculum developers but can be aided by securing detailed feedback from CTE teachers. Drawing on a recent study, this issue brief identifies challenges developers faced in infusing more math into CTE curricula and notes that incorporating academic learning into CTE requires substantial time, effort, and other resources.

Working Papers

• "Shrinkage of Value-Added Estimates and Characteristics of Students with Hard-to-Predict Achievement Levels." Mariesa Herrmann, Elias Walsh, Eric Isenberg, and Alexandra Resch. April 2013. This working paper investigates how empirical Bayes shrinkage, an approach commonly used in implementing teacher accountability systems, affects the value-added estimates of teachers of students with hard-to-predict achievement levels, such as students who have low prior achievement and receive free lunch. Teachers of these students tend to have less precise value-added estimates than teachers of other types of students. Shrinkage increases their estimates’ precision and reduces the absolute value of their value-added estimates. However, this paper found shrinkage has no statistically significant effect on the relative probability that teachers of hard-to-predict students receive value-added estimates that fall in the extremes of the value-added distribution and, as a result, receive consequences in the accountability system.
• "Does Tracking of Students Bias Value-Added Estimates for Teachers?" Ali Protik, Elias Walsh, Alexandra Resch, Eric Isenberg, and Emma Kopa, March 2013. This working paper uses urban school district data to investigate whether including track indicators or accounting for classroom characteristics in the value-added model is sufficient to eliminate potential bias resulting from the sorting of students into academic tracks. Accounting for two classroom characteristics—mean classroom achievement and the standard deviation of classroom achievement—may reduce bias for middle school math teachers, whereas track indicators help for high school reading teachers. However, including both of these measures simultaneously reduces the precision of the value-added estimates in this context. While these different specifications produce substantially different value-added estimates, they produce small changes in the tails of value-added distribution.
• "Horseshoes, Hand Grenades, and Treatment Effects? Reassessing Bias in Nonexperimental Estimators." Kenneth Fortson, Philip Gleason, Emma Kopa, and Natalya Verbitsky-Savitz, March 2013. Nonexperimental methods, such as regression modeling or statistical matching, produce unbiased estimates if the underlying assumptions hold, but these assumptions are usually not testable. Most studies testing nonexperimental designs find that they fail to produce unbiased estimates, but these studies have examined weaker evaluation designs. This working paper addresses these limitations and finds the use of baseline data that are strongly predictive of the key outcome measures considerably reduces bias, but might not completely eliminate it.
• "Is School Value-Added Indicative of Principal Quality?" Hanley Chiang, Stephen Lipscomb, and Brian Gill, November 2012. Using data on elementary and middle school math and reading outcomes for Pennsylvania students, this working paper found that school value-added provides little useful information for comparing the general leadership skills of different principals when these comparisons include some principals who are in their first three years at their current positions.
• "Student Selection, Attrition, and Replacement in KIPP Middle Schools." Ira Nichols-Barrer, Brian P. Gill, Philip Gleason, and Christina Clark Tuttle, September 2012. Using longitudinal, student-level data, this American Educational Research Association conference paper examines the entry and exit of students in KIPP middle schools, comparing KIPP's rates of attrition and replacement with rates in nearby district-run schools.
• "Statistical Power for Regression Discontinuity Designs in Education: Empirical Estimates of Design Effects Relative to Randomized Controlled Trials." John Deke and Lisa Dragoset, June 2012. Using data from four previously published education studies, this working paper finds that a study using a regression discontinuity design needs between 9 and 17 times as many schools or students as a randomized controlled trial to produce an impact with the same level of statistical precision. The need for a large sample is driven primarily by bandwidth selection, not adjusting for random misspecification error.
• "Methods for Accounting for Co-Teaching in Value-Added Models." Heinrich Hock and Eric Isenberg, June 2012. This working paper helps to address the issue of isolating the effect of each teacher on student achievement when the student is taught the same subject by more than one teacher. This paper considers and compares three methods—Partial Credit Method, Teacher Team Method, and Full Roster Method—to estimate teacher effects. Based on the analysis, the authors conclude that the latter two methods provide a more stable approach to estimating teacher effects on student achievement. Furthermore, the Full Roster Method offers the most promise for robust, practical implementation.
• "Assessing the Rothstein Test: Does It Really Show Teacher Value-Added Models Are Biased?" Dan Goldhaber and Duncan Chaplin, February 2012. This working paper illustrates—theoretically and through simulations—that the Rothstein falsification test is not definitive in indicating bias in value-added model estimates of current teacher contributions to student learning.

Other

• "Charter School Authorizers and Student Achievement." NCSPE Occasional Paper No. 219. Ron Zimmer, Brian Gill, Jonathon Attridge, and Kaitlin Obenauf, June 2013. This paper uses individual student-level data from Ohio–which permits a wide range of organizations to authorize charter schools—to examine the relationship between type of authorizer and charter-school effectiveness, as measured by students’ achievement trajectories.
• What Works Clearinghouse (WWC)
• Administered by Mathematica for the U.S. Department of Education's Institute of Education Sciences, the WWC produces a variety of reports that assess and summarize education research. WWC reports can help educators make important decisions about what curriculums to implement, what products to purchase, and what methods to use in their classrooms and schools.
  
  See the WWC's latest releases at whatworks.ed.gov and explore available Practice Guides, Intervention Reports, and Quick Reviews or take a guided tour of the site.