Education Policy Research

Reports

• "Impacts of Title I Supplemental Educational Services on Student Achievement." John Deke, Lisa Dragoset, Karen Bogen, and Brian Gill, May 2012. As part of No Child Left Behind, parents of low-income students in low-performing schools are offered Supplemental Educational Services (SES) for their children. These academic supports, such as extra tutoring or group sessions, take place outside the regular school day. This report for the Institute of Education Sciences examine potential achievement benefits. In the six study districts located in Connecticut, Florida, and Ohio, the program was directed to the lowest-achieving students due to oversubscription. However, not all students who were offered access to the program participated. The study found no evidence of impacts from offering SES to students near the cutoff of acceptance into the program. Furthermore, there were no impacts from participating in SES on student achievement in reading or math. Providers offered an average of 21 hours of SES per student for the school year, either one-on-one or in group sessions conducted by local teachers. No observed provider characteristics and practices, including intensity of services, were significantly associated with stronger impacts. The six districts were not nationally representative. Executive summary.
• "Using an Experimental Evaluation of Charter Schools to Test Whether Nonexperimental Comparison Group Methods Can Replicate Experimental Impact Estimates." Kenneth Fortson, Natalya Verbitsky-Savitz, Emma Kopa, and Philip Gleason, April 2012. Using data from Mathematica's experimental evaluation of charter schools, this methodological study examines the validity of four different comparison group approaches to test whether these designs can replicate findings from a well-implemented random assignment study.
• "Value-Added Models for the Pittsburgh Public Schools." Matthew Johnson, Stephen Lipscomb, Brian Gill, Kevin Booker, and Julie Bruch, February 2012. This report describes the value-added models (VAMs) created for the Pittsburgh Public Schools and the Pittsburgh Federation of Teachers. Pittsburgh's VAMs use not only state assessments but also course-specific assessments, student attendance, and course completion rates, aiming to produce estimates of the contributions of teachers and schools that are fair, valid, reliable, and robust.
• "Impacts of Performance Pay Under the Teacher Incentive Fund: Study Design Report." Steven Glazerman, Hanley Chiang, Alison Wellington, Jill Constantine, and Dan Player, October 2011. Mathematica's rigorous evaluation of the Teacher Incentive Fund (TIF) will estimate the impact of performance pay on student achievement, as well as educators' attitudes toward and awareness of TIF and their mobility and recruitment. This report provides contextual background on performance pay programs, characteristics of participating TIF grantee schools, and details on the TIF evaluation study design.
• "Moving Teachers: Implementation of Transfer Incentives in Seven Districts." Steven Glazerman, Ali Protik, Bing-ru Teh, Julie Bruch, and Neil Seftor, April 2012. By offering $20,000 per teacher, seven school districts piloting a transfer-incentive strategy, known as the Talent Transfer Initiative (TTI), filled 90 percent of their targeted vacancies in hard-to-staff schools with some of the districts' highest-performing teachers. A new study highlights the implementation experience and intermediate impacts of TTI, which is intended to expand disadvantaged students' access to the most effective teachers. Previous research conducted by Mathematica shows that, on average, low-income middle school students are significantly less likely to have access to the highest-performing teachers. Executive Summary
• "Learning from Charter School Management Organizations: Strategies for Student Behavior and Teacher Coaching." Robin Lake, Melissa Bowen, Allison Demeritt, Moira McCullough, Joshua Haimson, and Brian Gill, March 2012. A new Mathematica study, conducted with the Center on Reinventing Public Education, highlights approaches five successful charter school management organizations (CMOs) use to help improve student achievement. This report expands on a previous report showing that CMOs with the greatest positive impact on student achievement were most likely to establish consistent schoolwide behavior expectations for students, as well as use an intense approach to monitoring and coaching teachers. The latest report offers guidance for schools and districts looking to replicate these promising practices.
• "An Evaluation of the Chicago Teacher Advancement Program (Chicago TAP) After Four Years." Steven Glazerman and Allison Seifullah, March 2012. Mathematica's final report on the Chicago Teacher Advancement Program (Chicago TAP) found that the program did not raise student math or reading scores, but it increased teacher retention in some schools. For example, teachers in Chicago TAP schools at the start of the program in fall 2007 were about 20 percent more likely than teachers in comparison schools to be in those same schools three years later (67 percent versus 56 percent retention rate). However, the program did not have an impact on student achievement overall in the four-year rollout period in the Chicago Public Schools (CPS). Although Chicago TAP increased the amount of mentoring, promotion opportunities, and compensation in participating schools relative to non-TAP schools, the program did not fully implement its performance-based pay or value-added components as intended.

Journal Articles

• "Random Assignment Within Schools: Lessons Learned from the Teach For America Experiment." Steven Glazerman. Education Finance and Policy, April 2012 (subscription required). This article discusses the trade-offs associated with study designs that involve random assignment of students within schools and describes the experience from one such study of Teach For America (TFA). The article concludes that within-school random assignment studies such as the TFA evaluation are challenging but may also be feasible and generate useful evidence.
• "Examining Charter Student Achievement Effects Across Seven States." Ron Zimmer, Brian Gill, Kevin Booker, Stéphane Lavertu, and John Wittle. Economics of Education Review, April 2012 (subscription required). Previous charter school research has shown mixed results for student achievement, which could be the consequence of different policy environments or methodological approaches with differing assumptions across studies. This analysis discusses these approaches and assumptions and estimates effects using a consistent methodology across seven locations.
• "Links Between Young Children's Behavior and Achievement: The Role of Social Class and Classroom Composition." Annie Georges, Jeanne Brooks-Gunn, and Lizabeth M. Malone. American Behavioral Scientist, December 2011 (subscription required). This article examines the association between attentive and aggressive behavior (at the child- and class-level) and individual child achievement. Children with low attention, alone or in combination with aggressive behavior, made fewer gains in test scores during kindergarten. Additionally, having more children in the classroom with low attention was linked with lower achievement gains, even for children who did not themselves demonstrate problem behaviors.
• "The Effectiveness of Mandatory-Random Student Drug Testing: A Cluster Randomized Trial." Susanne James-Burdumy, Brian Goesling, John Deke, and Eric Einspruch. Journal of Adolescent Health, October 2011 (subscription required). This article presents findings from the largest experimental evaluation to date of school-based mandatory-random student drug testing (MRSDT) and its effectiveness in reducing substance use among high school students. The study found that students who were subject to MRSDT reported less substance use in the past 30 days than comparable students in schools without MRSDT.
• "Do Typical RCTs of Education Interventions Have Sufficient Statistical Power for Linking Impacts on Teacher Practice and Student Achievement Outcomes?" Peter Z. Schochet. Journal of Educational and Behavioral Statistics, August 2011(subscription required). For randomized controlled trials (RCTs) of education interventions, estimates of associations between student and mediating teacher practice outcomes can help examine the extent to which the data support the study's conceptual model. They can also identify mediators most associated with student learning. This article develops statistical power formulas for such exploratory analyses under clustered school-based RCTs using ordinary least squares and instrumental variable estimators and uses the formulas to conduct a simulated power analysis.
• "Using Lotteries to Evaluate Schools of Choice: Evidence from a National Study of Charter Schools." Christina Clark Tuttle, Philip Gleason, and Melissa Clark. Economics of Education Review, July 2011. This article draws on data and experiences observing and analyzing school lotteries from the National Evaluation of Charter School Impacts to describe challenges associated with lottery-based research. It presents evidence on the prevalence of oversubscribed charter middle schools and compares their features to charter schools that are not oversubscribed. It also describes the lotteries charter schools conduct and procedures for admitting students from waiting lists, as well as lessons for researchers for using lotteries to estimate impacts of schools on student outcomes.

Issue Briefs

• "Findings from a Randomized Experiment of Playworks: Selected Results from Cohort 1." Martha Bleeker, Susanne James-Burdumy, Nicholas Beyler, Allison Hedley Dodd, Rebecca A. London, Lisa Westrich, Katie Stokes-Guinan, and Sebastian Castrechini, April 2012. Findings based on the first cohort of schools included several significant, positive impacts. Playworks had a positive impact on teachers' perceptions of students' safety and feeling more included during recess. Teachers in Playworks schools also reported less bullying and exclusionary behavior during recess, and found transitions from recess to classroom learning were less difficult than teachers in control schools found. Teachers in Playworks schools reported significantly better student behavior at recess and readiness for class than teachers in control schools and were also more likely to report that their students enjoyed adult-organized recess activities. Students in Playworks schools reported better behavior and attention in class after sports, games, and play than students in control schools. Overall, most teachers, students, and principals reported positive perceptions of the Playworks program.
• "Do Low-Income Students Have Equal Access to the Highest-Performing Teachers?" Steven Glazerman and Jeffrey Max, April 2011. Most research on equal educational opportunity has focused on inputs like teacher experience and degrees. This brief estimated teachers’ value added (contribution to student achievement growth) and measured access to highest-performing teachers in high- and low-poverty schools. Across 10 selected districts in seven states students in the highest-poverty schools had unequal access, on average, to the district’s highest-performing middle school teachers. The pattern for elementary school was less clear. The degree of equal access varied by district. Technical Appendix.
• "Infusing Academics into Career and Technical Education." Trends in Education Research, Issue Brief #3. Joshua Haimson, James R. Stone, III, and Donna Pearson, December 2008. Integrating academic learning into career and technical education (CTE) classes can be challenging for educators and curriculum developers but can be aided by securing detailed feedback from CTE teachers. Drawing on a recent study, this issue brief identifies challenges developers faced in infusing more math into CTE curricula and notes that incorporating academic learning into CTE requires substantial time, effort, and other resources.

Working Papers

• "Assessing the Rothstein Test: Does It Really Show Teacher Value-Added Models Are Biased?" Dan Goldhaber and Duncan Chaplin, February 2012. This working paper illustrates—theoretically and through simulations—that the Rothstein falsification test is not definitive in indicating bias in value-added model estimates of current teacher contributions to student learning.
• "Do Charter Schools Improve Student Achievement? Evidence from a National Randomized Study." Melissa A. Clark, Philip Gleason, Christina Clark Tuttle, and Marsha K. Silverberg, December 2011. This paper looks at findings from the first national randomized study of the impacts of charter schools on student achievement.
• "False Performance Gains: A Critique of Successive Cohort Indicators." Steven M. Glazerman and Liz Potamites, December 2011. There are many ways to use student test scores to evaluate schools. This paper defines and examines different estimators including regression-based value-added indicators, average gains, and successive cohort differences in achievement levels. The paper helps assess which methods provide useful information for school accountability and why.
• "High School Dual Enrollment Programs: Are We Fast-Tracking Students Too Fast?" Cecilia Speroni, December 2011. This study tracked a subset of Florida's 2000–2001 and 2001–2002 high school seniors who took a college algebra placement test. Students who passed the test (with a score very near the cut-off for eligibility) and enrolled in a rigorous dual enrollment college algebra class were 16 percent more likely to go to college and 23 percent more likely to earn a college degree than similar students who did not take the class.
• "Determinants of Students' Success: The Role of Advanced Placement and Dual Enrollment Programs." Cecilia Speroni, November 2011. This study tracked all of Florida's 2000–2001 and 2001–2002 high school seniors and found that students who participated in dual enrollment classes on college campuses were 12 percent more likely to go to college and 7 percent more likely to earn a bachelor's degree than similar students who did not participate. Students who took these classes exclusively on the high school campus showed no statistically significant gains. The study found that dual enrollment and Advanced Placement participation also had similarly positive impacts.
• "Student Selection, Attrition, and Replacement in KIPP Middle Schools." Ira Nichols-Barrer, Christina Clark Tuttle, Brian P. Gill, and Philip Gleason, April 2011.Using longitudinal, student-level data, this American Educational Research Association conference paper examines the entry and exit of students in KIPP middle schools, comparing KIPP’s rates of attrition and replacement with rates in nearby district-run schools.

Other

• "Passing Muster: Evaluating Teacher Evaluation Systems." Steven Glazerman, Dan Goldhaber, Susanna Loeb, Stephen Raudenbush, Douglas O. Staiger, Grover J. Whitehurst, and Michelle Croft, April 2011. This report addresses the comparison of teacher evaluation systems and proposes ways to achieve a uniform standard for dispensing funds to districts to recognize exceptional teachers without imposing a uniform evaluation system on those districts. The report provides practical procedures to determine reliable local teacher evaluation systems. It also demonstrates how the reliability of the evaluation system determines the proportion of teachers who can be identified as exceptional.
• "Student Selection, Attrition, and Replacement in KIPP Middle Schools: Working Paper Presented at the 2011 Annual Meeting of the American Educational Research Association." Ira Nichols-Barrer, Christina Clark Tuttle, Brian P. Gill, and Philip Gleason, April 2011. Using longitudinal, student-level data, this American Educational Research Association conference paper examines the entry and exit of students in KIPP middle schools, comparing KIPP’s rates of attrition and replacement with rates in nearby district-run schools.
• "Estimating the Return to College Selectivity Over the Career Using Administrative Earning Data." Stacy Dale and Alan B. Krueger, February 2011. This paper uses administrative earnings data to the effect of attending a highly selective college on future earnings. It extends the work of a previous paper—that examined the 1995 earnings of a cohort of students who entered college in 1976. Using earnings data from a longer time horizon (from 1983 through 2007) as well as data for a more recent cohort of students who entered college in 1989, the researchers found that estimates of the return to college selectivity fell substantially, often to near zero. However, for black, Hispanic, and students from less-educated families (in terms of their parents’ education), the estimates of the return to college selectivity remained large.
• What Works Clearinghouse (WWC)
• Administered by Mathematica for the U.S. Department of Education's Institute of Education Sciences, the WWC produces a variety of reports that assess and summarize education research. WWC reports can help educators make important decisions about what curriculums to implement, what products to purchase, and what methods to use in their classrooms and schools.
  
  See the WWC's latest releases at whatworks.ed.gov and explore available Practice Guides, Intervention Reports, and Quick Reviews or take a guided tour of the site.