Get Updates
Reading Comprehension Curricula Show No Positive Impacts on Achievement, First-Year Findings of New Federal Evaluation Reveal
Contact: Cheryl Pedersen, (609) 275-2258
PRINCETON, N.J. (May 5, 2009)—To become successful learners, students need to comprehend what they read. Reading comprehension becomes increasingly important as students move into upper elementary grades, when they transition from “learning to read” to “reading to learn.” Children from disadvantaged backgrounds, in particular, may have difficulty comprehending text because they often lack general vocabulary and strategies for organizing information and gleaning knowledge from text.
A new federal study of four reading comprehension programs sheds light on the effectiveness of these curricula in helping disadvantaged students improve their reading comprehension. Findings released today reveal that, overall, these curricula had no positive impact on student test scores, and in some cases, had a negative impact. The study, a large-scale randomized control trial conducted by Mathematica Policy Research, Inc., for the U.S. Department of Education’s Institute of Education Sciences, examined the effects of these curricula on 5th grade students. It involved 268 teachers and 6,350 students in 89 schools in 10 mostly large disadvantaged urban districts in 8 states.
Mathematica evaluated four curricula that supplement schools’ core reading curriculum—Project CRISS, ReadAbout, Read for Real, and Reading for Knowledge (see last page for a description of the curricula). These curricula are designed to be used by schools and teachers to improve students’ comprehension skills.
As part of the study, one of the largest and most rigorous of its kind, schools were randomly assigned to one of four intervention groups or the control group. Researchers compared students in each intervention group with the control group. They also compared the combined group of all students receiving any of the curricula with the control group. Comparisons were based on a general reading comprehension test called the Group Reading Assessment and Diagnostic Evaluation (GRADE), tests of comprehension of science and social studies passages, and a composite score based on all the tests.
Main findings from the study include:
- Overall, average test scores in schools randomly assigned to use the four curricula were not statistically significantly higher than scores in control schools. There were no positive effects on the GRADE or on the science or social studies reading comprehension assessments. Furthermore, reading comprehension in schools assigned to use one of three commercially-available curricula (Project CRISS, ReadAbout, and Read for Real) was not significantly different from that of schools in the control group. Students in schools using the Reading for Knowledge curriculum scored statistically significantly lower than control group students on the science reading comprehension assessment and the composite score.
- For certain subgroups, test scores in schools using the selected curricula were significantly lower than scores in control schools, although there was no clear pattern to these findings. These negative effects were found for students in high-poverty schools and students with the lowest initial reading comprehension skills (bottom third). Negative results were also found for students with above-average initial reading fluency skills and students taught by educators with more than five years of experience.
- Over 80 percent of teachers reported using the intervention programs they were assigned to use, and, on average, teachers were observed to be adhering to between 61 and 78 percent of the specific components of the programs.
“While these programs were designed based on research about reading comprehension, until now they had not been rigorously evaluated on a large scale to determine whether they are effective,” said Susanne James-Burdumy, lead author and associate director of research at Mathematica’s New Jersey office. “This report is an important step in efforts to identify curricula that improve reading comprehension. Future reports will assess the extent to which positive impacts on students might develop over time, as well as whether these curricula are more effective after schools and teachers have had one year of experience using them.”
The report, “Effectiveness of Selected Supplemental Reading Comprehension Interventions: Impacts on a First Cohort of Fifth-Grade Students,” by James-Burdumy, Wendy Mansfield, John Deke, Nancy Carey, Julieta Lugo-Gil, Alan Hershey, Aaron Douglas, Russell Gersten, Rebecca Newman-Gonchar, Joseph Dimino, and Bonnie Faddis, presents the background and design of the evaluation and impact results from the 2006-2007 school year—the first year of intervention implementation and data collection. Baseline student assessment data were collected using the GRADE and the Test of Silent Contextual Reading Fluency (TOSCRF). A teacher survey was also administered at baseline. Follow-up data collection included the GRADE, science and social studies reading comprehension assessments developed by Educational Testing Service, and the collection of school and student records. Classroom observations assessed teacher instructional practices.
About the Curricula
The following text briefly describes the key elements of each curriculum examined in the study:
Project CRISS (developed by Creating Independence Through Student-Owned Strategies) focuses on five keys to learning—background knowledge, purpose setting, author’s craft (which involves using text structure to improve comprehension), active learning, and metacognition. The program is designed to be used each day during language arts, science, or social studies periods.
ReadAbout (developed by Scholastic Inc.) teaches students reading comprehension skills, such as author’s purpose, main idea, cause and effect, compare and contrast, summarizing, and inferences, primarily through a computer program. Students apply what they have learned to a selection of science and social studies trade books.
Read for Real (developed by Chapman University and Zaner-Bloser) supplies teachers with a six-volume set of books to teach reading strategies to students (for example, previewing, activating prior knowledge, setting a purpose, main idea, graphic organizers, and text structures). The books can be used before, during, and after reading. Each unit includes vocabulary, fluency, and writing activities.
Reading for Knowledge (created by the Success for All Foundation for inclusion in the study) makes extensive use of cooperative learning strategies and a process called SQRRRL (Survey, Question, Read, Restate, Review, Learn).
Mathematica, a nonpartisan research firm, conducts high-quality, objective policy research and surveys to improve public well-being. Its clients include federal and state governments, foundations, and private-sector and international organizations. The employee-owned company, with offices in Princeton, N.J., Ann Arbor, Mich., Cambridge, Mass., Chicago, Ill., Oakland, Calif., and Washington, D.C., has conducted some of the most important studies of education, health care, family support, employment, nutrition, and early childhood policies and programs.