Case Study: The Diverse Strategies and Student Impacts of Charter School Management Organizations

" . . . achievement impacts for individual CMOs are more often positive than negative, but vary substantially in both directions.”
- Brian Gill in Education Week
Project Facts

This study provided the first rigorous nationwide examination of CMOs’ effects on the critical long-term outcomes of students’ high school graduation and college entry rates. Researchers used multiple data sources to describe CMOs, assess their impacts on students, and identify practices associated with positive impacts. Key findings include: 

  • CMOs’ impacts on student achievement vary substantially: Many CMOs have a significant positive impact on students’ academic achievement as captured by test scores, and others have significant negative impacts.
  • Each CMO’s impact on test scores is typically consistent across schools, suggesting that CMOs are having some success in promoting uniformity (whether in a positive or negative direction).
  • Some—but not all—CMOs substantially boost students’ chances of graduating from high school and enrolling in postsecondary education. Of the six CMOs for which relevant high school data were available, three had significant positive impacts on students’ graduation rates. Of the four CMOs with data on students’ postsecondary enrollment, two had large positive impacts.
  • Some CMOs have implemented policies, programs, and procedures that enable them to systematically outperform other CMOs. CMOs with positive student achievement impacts have schools that emphasize two practices: (1) student behavior policies, and (2) intensive teacher coaching and monitoring.

Notes on the Study Design and Data Collection. The study team implemented a randomized controlled trial (RCT) and a larger quasi-experimental design (QED) to assess whether QEDs can replicate RCTs in the context of the study. They found that their QED approach could replicate an RCT, which then gave them greater confidence in findings from QEDs implemented in sites where RCTs were not feasible. To examine eligible CMOs and address the research questions, the study team conducted a survey of CMO central office staff, surveys of CMO principals and principals in nearby conventional public schools, a survey of CMO teachers, and site visits to 10 CMOs and 20 schools. In addition, they collected and analyzed school records with data on student characteristics and outcomes (including test scores) and examined CMOs’ financial records and business plans.

The Issue

CMOs operate multiple charter schools and create new schools under a common structure and philosophy, attempting to implement promising educational practices on a large scale. In recent years, the number and reach of CMOs have increased substantially, leading to a diverse array of organizational models and school designs. By 2012, there were at least 130 CMOs nationwide, accounting for about 16 percent of all charter schools. Some CMOs have received laudatory attention following reports of dramatic achievement results for disadvantaged students. Many of these reports, however, relied on incomplete evidence. Many educators, policymakers, and funders are interested in ways to identify and replicate successful charter schools and help other public schools adopt effective practices. This national study provided rigorous evidence on CMOs’ practices and impacts, identifying promising practices associated with positive impacts.

The Approach

To provide information on the variability of CMOs’ effectiveness, the study team estimated achievement effects separately for each CMO for which data were available. The study team also looked at the CMOs’ structures, practices, and policy contexts that might have influenced these outcomes. In addition, they examined the relationships between CMOs’ practices and their effects on student achievement. The impact analysis employed both experimental and quasi-experimental methods. Data sources included school and district records; site visits; interviews with CMOs’ school and central office staff members; CMO and district cost data; and surveys of CMOs, principals, and teachers.

The Impacts
The National Study of CMO Effectiveness provided the most detailed and rigorous examination of CMOs and their impacts to date. This research identified CMOs’ structures and practices that are most effective in raising achievement and documented substantial variation in CMOs’ impacts on students’ achievement and in CMOs’ use of particular educational strategies and practices. These findings inform the continuing debate on the effectiveness of charter schools and how best to structure them to achieve positive educational results. The study also identified areas for further research, including impacts on longer-term outcomes such as postsecondary enrollment, degree completion, civic behavior, and earnings and questions about implementation and why some CMOs had significant negative impacts on test scores. 
To learn more about how Mathematica Policy Research can provide solutions to your challenges, contact: Communications@mathematica-mpr.com.

This case study is for informational purposes only. Mathematica Policy Research, a nonpartisan research firm, provides a full range of research and data collection services, including program evaluation and policy research, survey design  and data collection, research assessment and interpretation, and program performance/data management, to improve public well-being. Its clients include federal and state governments, foundations, and private-sector and international organizations. The employee-owned company, with offices in Princeton, N.J.; Ann Arbor, Mich.; Cambridge, Mass.; Chicago, Ill.; Oakland, Calif.; and Washington, D.C., has conducted some of the most important studies of education, disability, health care, family support, employment, nutrition, and early childhood policies and programs.

About the Project

Commissioned by the NewSchools Venture Fund, with the support of the Bill & Melinda Gates Foundation and the Walton Family Foundation, the National Study of Charter Management Organizations’ (CMOs) Effectiveness was a longitudinal research effort designed to measure how nonprofit CMOs affect student achievement and examine the internal structures, practices, and policy contexts that can influence these outcomes. The study, conducted by Mathematica Policy Research and the University of Washington’s Center on Reinventing Public Education, began in May 2008 and concluded in 2012.