|
Education: Latest Work
Reports | Journal Articles | Issue Briefs | Other
Reports |
|
|
| "Guidelines for Multiple Testing in Impact Evaluations of Educational Interventions." Peter Z. Schochet, May 2008. Statistical procedures that correct for multiple testing typically result in hypothesis tests with reduced statistical power because adjustment methods reduce the likelihood of identifying real differences between contrasted groups. There is disagreement among researchers about the use of multiple testing procedures and the appropriate trade-off between type I error and statistical power (type II error). These guidelines were developed to handle multiple testing in education research. In addition, the report provides details on the nature of the multiple testing problem and the statistical solutions that have been proposed; the creation of composite outcomes measures; and the Bayesian hypothesis testing approach. |
|
| "The Nation's Report Card: Technical Report of the NAEP Mathematics Assessment in Puerto Rico: Focus on Statistical Issues." G.P. Baxter, S. Ahmed, E. Sikali, T. Waits, M. Sloan, and S. Salvucci, September 2007. In 2003, a trial National Assessment of Educational Progress mathematics assessment was administered in Spanish to public school students in grades 4 and 8 in Puerto Rico. On the basis of preliminary analyses of the 2003 data, changes were made in administration and translation procedures for 2005. This report describes the content and administration of the trial assessments in Puerto Rico in 2003 and 2005, problems with item misfit in the 2003 data, results of a special validity analysis, and plans to integrate Puerto Rico into the national sample in future administrations. |
|
| "Evaluation of the Teacher Advancement Program (TAP) in the Chicago Public Schools: Study Design Report." Steven Glazerman, Allison McKie, Nancy Carey, and Dominic Harris, November 2007. Recent evidence has confirmed that teacher quality is a critical component in student achievement. The Teacher Advancement Program (TAP) is a whole-school approach to evaluating and compensating teachers and providing professional development opportunities to both improve teaching and help schools attract and retain good teachers. This report describes Mathematica’s five-year evaluation, which began in 2007, in high-need Chicago public schools. |
|
| “Measuring School Effectiveness in Memphis.” Kevin Booker and Eric Isenberg, April 2008. New Leaders for New Schools, a nonprofit organization committed to training school principals, is heading an initiative that offers financial awards to effective educators. This report details the value-added model Mathematica developed to identify effective schools in the Memphis City school district during the first year of the project. The model estimates each school’s effect on student performance across all tested grades and subjects, accounting for student mobility, observable differences in student demographics, and measurement error in test scores. |
|
| "Achievement and Attainment in Chicago Charter Schools." Kevin Booker, Brian Gill, Ron Zimmer, and Tim R. Sass, May 2008. During the past decade, charter schools have been one of the fastest growing segments of the K–12 education sector. This is the first report to examine how charter schools may affect long-term attainment, including high-school graduation and college entry. The authors find that Chicago's multi-grade charter high schools (those that include grades 6-12, 7-12, or K-12) appear to produce substantial positive effects on ACT scores, probability of graduating, and probability of enrolling in college. The large positive results suggest promise for multi-grade charter high schools (and perhaps charter high schools more generally) and demonstrate that evaluations limited to test scores may fail to capture important benefits of charter schools. |
|
| "Title I School Choice and Supplemental Educational Services: Interim Report." Brian Gill, Jennifer Sloan McCombs, Scott Naftel, Karen Ross, Mengli Song, Jennifer Harmon, and Georges Vernez. State and Local Implementation of the No Child Left Behind Act: Volume IV, 2008. This report presents findings on the implementation of parental choice options from the first year of two national studies of No Child Left Behind. In 2004-2005, nearly 6.2 million students were eligible for Title I school choice and as many as 1.8 million were eligible for supplemental educational services. Only about one percent took advantage of the school choice option, and about 17 percent took advantage of supplemental services. In a subsample of large urban districts, the average achievement of the schools chosen by students using the school choice option was consistently higher than the average achievement of the schools they came from, and their parents were largely satisfied with the new schools. The report also notes that low participation rates in Title I school choice and supplemental educational services may be related to problems communicating with parents. |
|
|
"National Assessment of Title I Final Report. Volume II, Closing the Reading Gap: Findings from a Randomized Trial of Four Reading Interventions for Striving Readers." Joseph Torgesen, Allen Schirm, Laura Castner, Sonya Vartivarian, Wendy Mansfield, David Myers, Fran Stancavage, Donna Durno, Rosanne Javorsky, and Cinthia Haan, October 2007. This report details findings from an experimental evaluation of four widely used programs for elementary school students with reading problems—Corrective Reading, Failure Free Reading, Spell Read P.A.T., and Wilson Reading. Conducted just outside Pittsburgh, Pennsylvania, in the Allegheny Intermediate Unit, the evaluation explored the extent to which the programs affect both word-level reading and reading comprehension skills of students in grades three and five identified as struggling readers. This report presents impacts on reading test scores one year after the interventions ended, when most of the students were in fourth and sixth grade, as well as impacts on state reading and mathematics test scores from the previous year. Key findings show that the interventions improved some reading skills but did not improve state test scores. Furthermore, younger students benefited more, and the interventions narrowed some reading gaps. Appendices |
|
|
"The Postsecondary Achievement of Participants in Dual Enrollment: An Analysis of Student Outcomes in Two States.” Melinda Mechur Karp, Juan Carlos Calcagno, Katherine L. Hughes, Dong Wook Jeong, and Thomas R. Bailey, October 2007. Dual enrollment programs enable high school students to enroll in college courses and earn college credit. Once limited to high-achieving students, such programs are increasingly seen as a means to support the postsecondary preparation of average-achieving students. Moreover, though dual enrollment programs typically have been reserved for academically focused students, increasing numbers of career and technical education (CTE) programs are providing such opportunities to their students. In this report, the authors examine the impact of dual enrollment participation for students in Florida and New York City, examining postsecondary outcomes for participating CTE students in both locations and outcomes of participation for all students in Florida. The report notes that dual enrollment is a useful strategy for encouraging postsecondary success for all students, including those in CTE programs. |
|
| Rhetoric vs. Reality: What We Know and What We Need to Know About Vouchers and Charter Schools. Brian Gill, P. Mike Timpane, Karen Ross, Dominic Brewer, and Kevin Booker, August 2007. Vouchers and charters are two of the most ambitious attempts to improve school performance through parental choice and privatization of governance. The updated edition of this book (which was first released in 2001) summarizes new empirical evidence on the achievement impacts of voucher programs and charter schools operating in various locations across the country (and, to some extent, around the world). It concludes that neither the hopes of the supporters nor the fears of the skeptics have been realized, but that well-designed voucher and charter programs have shown promising signs, while few have shown any evidence of substantial harms to student achievement. The authors note that the details of policy design are likely to be critical to the effects produced by choice programs: not all voucher programs and charter laws are alike, and different programs will produce different effects. They call for additional research to help clarify the importance of specific policy elements and to assess a deeper and wider range of student outcomes. |
|
Journal Articles |
|
|
| “Statistical Power for Random Assignment Evaluations of Education Programs.” Peter Z. Schochet, Journal of Educational and Behavioral Statistics, March 2008. This article examines theoretical and empirical issues related to the statistical power of impact estimates for experimental evaluations of education programs. The author considers designs where random assignment is conducted at the school, classroom, or student level, and employs a unified analytic framework using statistical methods from the literature. Focusing on standardized test scores of elementary school students, this article discusses appropriate precision standards and, for each design, the required number of schools to achieve those standards using empirical values of intraclass correlations, regression R2 values, and other parameters. Clustering effects vary by design but are typically large. As a result, large school samples are required for education trials, and many evaluations will have sufficient power to detect precise impacts only for relatively large subgroups of sites. |
|
| “After-School Program Effects on Behavior: Results from the 21st Century Community Learning Centers Program National Evaluation.” Susanne James-Burdumy, Mark Dynarski, and John Deke, Economic Inquiry, January 2008. This paper presents evidence on after-school programs’ effects on behavior from the national evaluation of the U.S. Department of Education’s 21st Century Community Learning Centers after-school program. Findings come from both of the study’s components: (1) an elementary-school component based on random assignment of 2,308 students in 12 school districts, and (2) a middle-school component based on a matched comparison design including 4,264 students in 32 districts. Key findings include higher levels of negative behavior for elementary students and some evidence of higher levels of negative behaviors for middle school students. |
|
| "Academic Achievement and School Functioning Among Non-Incarcerated Youth Involved with the Juvenile Justice System." Jonathan D. Brown, Anne W. Riley, Christine M. Walrath, Philip J. Leaf, and Carmen Valdez, Journal of Education for Students Placed at Risk, January-March 2008. This article reports on the education problems of youth involved with the juvenile justice system but not incarcerated. More than half demonstrated deficits in academic functioning, with standard achievement scores as low as five standard deviations below the normative mean. Non-Caucasian youth and those who received special education services or lived in an urban area had lower achievement. These findings suggest that youth involved with the justice system but not incarcerated demonstrate problems in academic achievement similar to incarcerated youth and may benefit from targeted education interventions. |
|
|
"Governing the Family Through Education: A Genealogy on the Home/School Relation." Kirsten Kainz and Nikki L. Aikens, Equity & Excellence in Education, October 2007. The authors examine a century of rhetoric on the home/school relation and associated parent involvement. The review investigates how current ideas about this relationship and parent involvement might inhibit the realization of justice, equity, and excellence in education. |
|
| "When Schools Stay Open Late: Results from the National Evaluation of the 21st Century Community Learning Centers Program." Susanne James-Burdumy, Mark Dynarski, and John Deke, Educational Evaluation and Policy Analysis, December 2007. This article presents evidence from the elementary school component of Mathematica’s national evaluation of the 21st Century Community Learning Center afterschool programs. The findings indicate that the programs affected the supervision students received after school, with parents less likely to care for their child and other adults more likely, but there was no statistically significant effect on the incidence of self-care. Students in the program reported feeling safer after school, but their academic outcomes were not affected, and they had more incidents of negative behavior. The elementary study was conducted in 12 school districts and 26 afterschool centers. |
|
|
"Stepping Stones to a Degree: The Impact of Enrollment Pathways and Milestones on Community College Student Outcomes." Juan Carlos Calcagno, Peter Crosta, Thomas Bailey, and Davis Jenkins, Research in Higher Education, November 2007. This paper presents findings from a study of the experiences and outcomes of older and younger community college students. The authors developed a discrete-time hazard model using longitudinal transcript data on a cohort of first-time community college students in Florida to compare the impact of enrollment pathways (such as remediation) and enrollment milestones (such as attaining a certain number of credits) on educational outcomes of older students—those who enter college for the first time at age 25 or later—with those of traditional-age students. Results suggest that reaching milestones such as obtaining 20 credits or completing 50 percent of a program is a more important positive factor affecting graduation probabilities for younger students than it is for older students. |
|
Issue Briefs |
|
"Expanding Beyond Academics: Who Benefits and How?" John Deke and Joshua Haimson, September 2006. The growing use of math and reading scores to measure school and student performance, spurred by the No Child Left Behind Act, has heightened an old debate about which competencies public schools should encourage students to develop. This issue brief looks at how students' competencies in high school relate to postsecondary educational attainment and earnings later in life. These competencies include academic achievement (as measured by test scores), leadership skills, sports-related skills, work habits, prosocial behavior, and locus of control (a measure of students' belief that they control their future). The brief concludes that many of these competencies appear to predict students' later success in higher education and the labor market. It also concludes that not all students would benefit from improving the same competencies, suggesting that an individualized approach to education may be preferable to a "one-size-fits-all" approach.
Full Report |
|
Other |
|
| The What Works Clearinghouse (WWC), which is administered by Mathematica for the Institute of Education Sciences, released a new intervention report in the area of dropout prevention. Accelerated Middle Schools, self-contained academic programs to help middle school students who are one to two years behind grade level catch up to their peers, give students the opportunity to cover an additional year of curriculum during their one to two years in the program. Programs can be structured as separate schools or as schools within a traditional middle school. The WWC found accelerated middle schools to have potentially positive effects on staying in school and positive effects on progressing in school. |
|
The What Works Clearinghouse (WWC), which is administered by Mathematica for the Institute of Education Sciences, released a new type of evidence report. The WWC Quick Review provides an objective assessment of the quality of the research evidence from a research paper or report whose public release is reported in a major national news source. The WWC Quick Review assesses whether the research described in the paper or report is consistent with WWC evidence standards.
New WWC Quick Reviews include:
|
|
| What Works Clearinghouse practice guide: Turning Around Chronically Low-Performing Schools, May 2008. The guide identifies practices that can improve the performance of chronically low-performing schools—a process commonly referred to as creating "turnaround schools." The four recommendations in the guide aim to help failing schools make adequate yearly progress. |
|
| “Evaluating the Performance of Philadelphia’s Charter Schools.” Ron Zimmer, Suzanne Blanc, Brian Gill, and Jolley Christman, March 2008. Charter schools are publicly funded schools that operate autonomously, outside the direct control of local school districts, and generally enroll students through the choices of their parents rather than through residential assignment. These schools have proved popular nationwide. The authors examine effects of charter schools on reading and mathematics achievement for students in Philadelphia. They also examine effects of years of operation, grades served, mission, and demographics on student achievement; types of students charter schools attract; turnover rates; and charter schools’ impact on student achievement in traditional public schools. Achievement gains of students attending charter schools are approximately equal to the gains of students attending district-operated public schools. There is little evidence that charter schools either help or harm the achievement of students in nearby district schools. |
|
Two new What Works Clearinghouse reviews in the area of dropout prevention are available. Conducted by a team led by principal investigator Robert Wood, the reviews focus on interventions designed to help disadvantaged youth complete their education and prepare for work.
“What Works Clearinghouse Intervention Report: JOBSTART.” An alternative education and training program, JOBSTART is designed to improve the economic prospects of young, disadvantaged high school dropouts. The WWC found this intervention to have potentially positive effects on completing school.
"What Works Clearinghouse Intervention Report: Job Corps.” A program for economically disadvantaged youth, Job Corps offers remedial education, GED preparation, vocational training, job placement assistance, and other supports. The WWC found Job Corps to have no discernible effects on progressing in school and potentially positive effects on completing school. |
|
| “Missouri’s Teacher Career Ladder Program.” Kevin Booker and Steven Glazerman, February 2008. Although Missouri has had a Career Ladder program for teachers since 1987, there has been little research examining the program’s effects. This working paper, presented at a National Center on Performance Incentives conference, examines effects on student achievement, using longitudinal data on district math and reading scores for 524 Missouri school districts over a nine-year period. The paper compares achievement levels in participating districts with those of a matched group of nonparticipating districts. The estimated effects range from small positive effects to no effect. The authors conclude that, if the Career Ladder has a positive effect on test scores, it is probably very small. A soon-to-be-completed companion paper will explore program operations, and another will examine the relationship between Career Ladder participation and teacher retention. |
|
| "School Competition and Student Outcomes." Brian Gill and Kevin Booker. In Handbook of Research in Education Finance and Policy, 2008. A variety of policies can promote market forces in education, potentially producing competitive effects. These effects—positive or negative—are among the most important outcomes produced by educational privatization and school choice. This chapter addresses issues related to the effects of competition on conventional public schools and the traditional public purposes of education, including student integration and education of citizens.
To find all education-related publications, please go to the Publications Search page.
|
|
Back to Top |
