Overview of the Early Head Start Research and Evaluation Project

The Quiet Crisis
Structure of the Study
Evaluation Team
Implementation Analysis
Impacts
Outcomes
Measures

The Early Head Start Research and Evaluation project launches an intense study of the new Early Head Start program and simultaneously begins a far-reaching longitudinal study of infants and toddlers in low-income families. The Early Head Start study will include approximately 3,000 families living in 17 diverse communities that reflect the socioeconomic and political context of low-income families in the United States in the late 1990s. We will measure a broad range of outcomes, collect extensive information about the programs and the individual families’ experiences with them, and conduct state-of-the-art analyses to link experiences with outcomes. The findings from this national evaluation and longitudinal study will have the potential to influence policies affecting the lives of low-income American families with young children.

Responding to the "Quiet Crisis"

Early Head Start comes at a time of increasing awareness of the "quiet crisis" facing families with infants and toddlers in the United States, as identified in Carnegie Corporation of New York, Starting Points report. The Administration on Children, Youth and Families (ACYF) designed the Early Head Start program in response to the 1994 Head Start reauthorization, which established a special initiative setting aside three percent of 1995 Head Start funding, four percent of 1996 and 1997 funding, and five percent of 1998 funding for services to families with infants and toddlers. Secretary Shalala’s Advisory Committee on Services for Families with Infants and Toddlers then set forth a vision and blueprint for Early Head Start programs to address the fragmentation of community services and expand programs to serve more families with infants and toddlers. This comprehensive, two-generation program includes intensive services that begin before the child is born and concentrate on enhancing the child’s development and supporting the family during the critical first three years of the child’s life.

Early Head Start programs are designed to produce outcomes in four domains: (1) children’s development (including health, resiliency, social competence, and cognitive and language development), (2) family development (including parenting and relationships with children, the home environment and family functioning, family health, parent involvement, and economic self-sufficiency), (3) staff development (including professional development and relationships with parents), and (4) community development (including enhanced child care quality, community collaboration, and integration of services to support families with young children). Sixty-eight local programs, serving about 75 families each, were funded in September 1995, 75 in September 1996, and 32 in September 1997. Grants in 1998 brought the total number of programs to more than 290, and in early 2001, some 650 programs served more than 55,000 families. From the first two waves of programs, 17 were selected to participate in the national evaluation; 16 of these will participate in additional site-specific research studies.

The programs participating in the national evaluation and local research represent a wide diversity of locations, populations, culture, ethnicity, and urban-rural settings. They are located in Russellville, Arkansas; Venice, California; Denver, Colorado (two programs); Marshalltown, Iowa; Kansas City, Kansas; Jackson, Michigan; New York City; Kansas City, Missouri; Pittsburgh, Pennsylvania; Sumter, South Carolina; McKenzie, Tennessee; Logan, Utah; Alexandria, Virginia; Kent, Washington; Sunnyside, Washington; and Brattleboro, Vermont.

Structuring a National Research and Evaluation Project

A comprehensive national evaluation conducted in tandem with a cluster of local research and evaluation studies will address a broad range of issues. This research and evaluation will go beyond assessing program impacts to encourage a new generation of research for understanding the role of program and contextual variations, and it will create the foundation for a series of longitudinal research studies. To achieve this aim, the Early Head Start Research and Evaluation Project encompasses five major components:  

• An implementation study to examine service needs and use for low-income families with infants and toddlers, assess program implementation, understand programs’ theories of change, illuminate pathways to achieving quality, examine program contributions to community change, and identify and explore variations across sites
  
• An impact evaluation to analyze the effects of Early Head Start programs on children, parents, and families in depth, using an experimental design; descriptive analyses will assess outcomes for program staff and communities
  
• Local research studies by local researchers to learn more about the pathways to desired outcomes for infants and toddlers, parents and families, staff, and communities
  
• Policy studies to respond to information needs in areas of emerging policy-relevant issues, including welfare reform, fatherhood, child care, and children with disabilities
  
• Formats for continuous program improvement to guide all EHS programs in formative evaluation

Creating a Strong Evaluation Team

Following a competitive procurement process, ACYF contracted with Mathematica and its subcontractor, the Center for Young Children and Families (CYCF) at Columbia University, Teachers College. Mathematica and CYCF both have a national reputation for the quality of their work and their commitment to working with program operators and policymakers to plan and implement research and evaluations that address central policy issues for low-income families and their young children. Dr. John M. Love and Dr. Ellen Eliason Kisker of Mathematica and Dr. Jeanne Brooks-Gunn of Columbia University are leading the national evaluation team. The evaluation team also draws on expertise from a Technical Work Group and other consultants in key areas related to Early Head Start program and evaluation issues.

ACYF has also funded 15 local research teams to work with 16 of the Early Head Start research programs to conduct their own research on issues central to the local programs and to participate in many national evaluation activities (including providing field support for the national data collection). An Early Head Start Research Consortium composed of ACYF staff, national evaluation contractor staff, and the local researchers has been created to facilitate collaboration on issues relating to policy, assessment, and the use of research and evaluation data. Together, national and local research comprise the overall Early Head Start Research and Evaluation Project.

Describing and Analyzing Early Head Start Implementation

The implementation study is designed to (1) provide critical information on program implementation and context during the evaluation period; (2) identify and explore variations across programs, including variations in programs’ theories of change; (3) illuminate pathways to service quality; (4) examine service needs and use for low-income families with infants and toddlers; and (5) investigate program contributions to community change. The implementation study will provide critical information for designing specific impact analyses, interpreting impact evaluation findings, and deriving implementation lessons that will be useful for current and future programs. Information from the implementation study will assist ACYF in promoting continuous program improvement among Early Head Start programs.

Data for the implementation study will come from multiple sources, including three rounds of intensive site visits to the research programs, program documents, parent services follow-up interviews, child care observations, staff surveys, and other documents and databases. We will synthesize information from various sources according to established qualitative analysis methods and systematic procedures established in advance. We will create ratings of program implementation and service quality in the child development cornerstone by (1) gathering information on key implementation and quality criteria based on Head Start Performance Standards and previous research studies, and (2) convening a panel of experts to review the information and rate each program. Quantitative data on families’ service needs and use will be analyzed using appropriate descriptive statistical methods.

Studying Early Head Start Impacts on Children, Parents, and Families

A true experimental design is critical for a sound impact analysis. As programs recruit their families, we are randomly assigning them to the Early Head Start program group or a comparison group. Comparison group families will not be eligible for Early Head Start services but can receive all other services available in the community. We have worked with local programs and their research partners to design random-assignment procedures that are sensitive to program concerns while attending to possible threats to the integrity of the design.

To measure child and family outcomes, the consortium has chosen a comprehensive array of measures that will support a rich impact evaluation and also permit investigations by local researchers and the broader research community into a wide range of hypotheses, both in the short term and longitudinally. The measurement plan relies on proven, state-of-the-art instruments and procedures, yet it remains open to promising innovations. We are especially sensitive to the need for measures that are appropriate for the racial/ethnic and cultural diversity of Early Head Start program participants. An annual review of measures provides the opportunity to adapt measurement plans on the basis of new measures that become available. We use multiple data collection methods: parent reports, direct assessments of children, observations by trained observers, and coding of videotaped parent-child interactions in problem solving and free-play situations.

We are training local staff to conduct assessments with children at their 14-, 24-, and 36-month birth dates and interview parents at the same times to obtain information about family functioning and their children’s development. We measure parents’ use of services and children’s health at 6, 15, and 24 months after enrollment to ensure that information for comparison group families is comparable to program data on Early Head Start families. In addition, we ask parents about progress toward economic self-sufficiency, involvement in the program, and family health. During the 36-month birthday assessment, we also conduct an exit interview.

Analytic models that include growth curve analysis, hierarchical linear models, and adjustments for selection bias will enable us to (1) examine the overall impact of Early Head Start on children and families at two levels--for those who are randomly assigned and for those who actually receive program services; (2) assess differential effects for families with certain characteristics living in particular contexts; (3) explore differential impacts related to differences in program implementation, theories of change, and quality across sites; and (4) investigate how within-program variations in services received affect child and family outcomes.

Studying Outcomes for Program Staff and Communities

Early Head Start programs make staff development an integral part of their ongoing program improvement plans and forge strong relationships with other community-based agencies. Thus, the evaluation will systematically measure the most important outcomes for staff and communities. For staff, we will assess (1) their professional development, continuity, and health (through surveys of Early Head Start program directors and of key staff working with children and families); and (2) staff-parent relationships, which may influence the success of parenting education efforts (through parent surveys). For communities, we will gather qualitative information about the process of building relationships among families and service providers and building collaborative service networks.

In addition, because Early Head Start programs are also expected to influence the quality of services in the community, we have singled out child care--the most important community service for children--for special investigation. We are conducting observations of the quality of both Early Head Start and comparison group children’s nonparental child care arrangements when children are 14, 24, and 36 months of age. This study of child care will provide an assessment of infant, toddler, and preschool child care available to low-income families in the community over the entire period of the study. It will also provide important information on children’s environments and their relationships with caregivers for the impact study of families and children.

Producing Policy- and Program-Relevant Reports

We will document all plans, designs, procedures, and results of the national evaluation in periodic reports. So that the story of Early Head Start will benefit from both local and national perspectives, we will incorporate reports of local research activities at appropriate times. Relevance and timeliness are the central principles guiding our proposed reports. For the evaluation to be useful, its findings must be accessible. We will direct our reports to audiences of federal policymakers, local program staff, and the research community. In a series of reports, we will integrate findings of local researchers with cross-site data to present:

• A descriptive implementation report (1999)
  
• An interim implementation and program quality report (1999 and 2000)
  
• A final report on pathways to program quality (2001)
  
• A summary interim report (January 2001)
  
• An interim evaluation report (June 2001)
  
• Policy reports that present special analyses of important policy topics (2001-2002)
  
• A final program impact report (June 2002)

Unique Features of the EHS Research and Evaluation Project for Assessing Impacts of an Infant-Toddler Intervention

Early Head Start research and evaluation has benefited from earlier generations of intervention studies, including the evaluation of the Comprehensive Child Development Program (CCDP), and has been able to incorporate many methodological lessons from those studies. Several unique features will enable the Early Head Start study to answer questions many previous evaluations have been unable to address. Seven unique features are described here.

Identification of Effects as a Function of Program Variation. Early Head Start program approaches vary in important ways. For example, some programs deliver child development services through child care and others use a home visitation approach. These are appropriate alternatives in the context of different community needs, populations, and geography. The early intervention literature suggests that these variations will result in different patterns and timings of effects. Effects are also likely to differ because programs have different emphases; different theories of change and program approaches lead to different pathways by which their outcomes will be achieved.

Assessment of Implementation. Programs will be rated according to how well they implement the Head Start Performance Standards, using a consensus-based rating system in which a panel of experts reviews implementation data obtained from evaluation site visits. (The Advisory Committee on Services for Families with Infants and Toddlers recommended taking level of program implementation into account.) Only by systematically assessing program implementation against the Performance Standards is it possible to determine whether the program is being carried out in each site as it was designed.

Assessment of the Quality and Quantity of Child Development Services. This study will assess both the quality and the quantity of key services for children, including child care and home visits. In an age where child care is noted to be largely poor in quality, it is important to know the quality (as well as the quantity) of child care received by both the program and comparison group families. It is also important to measure the degree of child development emphasis in home visits and to create an understanding about the quality and quantity of home visits. Low-quality, infrequent visits that lack a child development focus would not be expected to have an impact on children, but high- quality, frequent visits oriented toward child development would be expected to have a positive impact on child development. Similarly, large doses of poor-quality care or small doses of high- quality care would not be expected to have an impact on children, but moderate to large amounts of high-quality child care would be expected to have a positive impact. In general, previous evaluations have not examined in detail the effects on children in light of different configurations of these child development service variables.

Assessment of Services Received by Both the Comparison Group and the Program Group. We know that the availability of services varies from community to community. For example, some communities are strong in health services but lack employment opportunities. Because different services are available to and actually used by the program and comparison groups, different patterns of outcomes would be expected in specific sites. We will assess community services available to the comparison group to make predictions on a site-by-site basis about the availability of resources within the community. We also are measuring directly the service use by both comparison and program groups, which will provide family-level data on service use for the impact analysis.

Analysis of Outcomes by Actual Program Participation. Program directors often state that they do not expect the program to affect the families who drop out. They expect impacts on those families with whom they had consistent and continuous contact over a period of several years. We will conduct rigorous impact analyses comparing all families randomly assigned to program and comparison groups; then, we will make statistical adjustments for the different levels of program participation. For example, we will estimate the program impacts on those children and families who received the full program dosage.

Implementation and Impact Study by the Same Research Team. In many program evaluations, the implementation and the impact studies are conducted by two different research organizations. This approach makes it difficult for the impact researchers to interpret site-level effects or to effectively use program implementation data in the impact analysis. In the Early Head Start study, these functions are being carried out by the same organization; this will facilitate the use of program process data in the impact analysis.

Partnerships with Local Researchers. Too often, local program interpretations are missing from national evaluations. A researcher with knowledge of the local program and community will be able to add interpretations to local findings that might not be apparent at the national level. Given that we expect site-level variation in patterns of effects, the project will benefit from local researchers’ further interpretation of site-specific findings in a way that will be useful to the programs.

Timing of the Early Head Start Evaluation

One might argue that the Early Head Start evaluation is being conducted too soon to detect strong program impacts. However, there is much to be learned about the early implementation of new programs as they face multiple challenges in moving to new approaches for serving low-income families with infants and toddlers and in adapting program models in response to welfare reform and the changing needs of families within Early Head Start communities. The dynamic state of new Early Head Start programs may be a factor in their early effectiveness, especially in the context of welfare reform. However, a strong, multifaceted evaluation with both implementation and impact analyses is the correct choice for facilitating continuous improvement from the lessons learned. A second wave of equally vigorous studies within established sites is an obvious followup. The lessons from the current evaluation will be invaluable for informing successive new waves of Early Head Start programs about alternative models and pathways for achieving program quality and impacts under various conditions. Frequent and timely reporting from this evaluation is our strategy for best informing the development of the new program.