Family outcomes: Head Start is a two-generation program that seeks to drive and sustain improvements in child outcomes by engaging parents to support their children’s learning and development. A meaningful system of differentiating program performance should include measures of family outcomes or changes in parent behavior that are associated with improved outcomes for children.
» Leading indicators: Leading indicators are intermediate outcomes that have a strong predictive relationship with ultimate outcomes of interest or reflect short-term outputs that lead to longer-term outcomes. Child attendance, for example, is both a strong predictor of children’s learning outcomes and an outcome of family engagement efforts.
» Safety, health, and compliance data: Even in a more outcomes-oriented system, there will still be some minimum input and compliance requirements — such as those related to health and safety or compliance with civil rights laws — that all programs must meet. These minimum standards should continue to be included in any performance measurement system but should indicate the floor, not the ceiling, of program performance. Further, system designers should also consider where outcomes — such as injuries or reported safety incidents — could potentially replace or supplement more compliance-oriented measures.
» Program quality data: Moneyball principles argue that, where possible, performance measurement and accountability systems should focus on results rather than inputs. Public policies, however, must also acknowledge that many factors outside of programs themselves can influence child and family outcomes in a given time period. Therefore, outcome measures should be complemented by measures of program quality and practices.
» Grantee health and capacity data: Producing strong outcomes requires effective organizations that use resources well, support and develop their staff, and use data to inform continuous improvement. While programs should have broad flexibility in how they choose to carry out these activities, a well-designed system of performance management should include measures of the organizational health and financial well- being needed to support program delivery.
Knowing how to measure program performance in some of these domains is challenging, however.
For example, Head Start grantees are required to establish goals for children’s school readiness and to collect child-level assessment data on their progress toward those goals.20 But the child assessments that most grantees use were primarily designed to inform teachers’ instruction and individualize learning for each child — not to provide information about the performance of Head Start programs.
Researchers have developed a number of promising, evidence-based tools to measure children’s learning in a variety of domains — including language, literacy, and science21 — and have validated or are currently validating these assessments. But these tools have not yet been widely implemented in Head Start or early childhood settings, or validated for large samples of children reflecting the Head Start population. Moreover, many of these tools focus on children’s development in a single domain, meaning that a comprehensive picture of children’s development outcomes would require using multiple assessments.
MONEYBALL FOR HEAD START 18
Similarly, researchers have developed a variety of promising new tools to measure quality in early childhood programs. Traditionally, early childhood programs have measured quality in terms of inputs, such as teacher credentials, square footage, and adult-child ratios. In contrast, the newer tools provide a systematic way of observing children’s actual experiences in early childhood programs and are more predictive of children’s learning than are traditional input measures.22 The Classroom Assessment Scoring System, used to measure the quality of adult- child interactions in Head Start classrooms, is one example. But, since the 2007 Head Start reauthorization, researchers have developed and validated, or are currently validating, a variety of other tools that look at children’s experiences, instructional quality, and other dimensions of program practice correlated with children’s learning.23
The emergence of new, evidence-based measures of both child learning and program quality creates the potential to develop much more informative and robust measures of Head Start performance. But many of these measures are very new, and a great deal of work remains to determine whether they should be implemented at scale for performance measurement purposes. Even after new assessment tools are vetted for validity and reliability, they must still be piloted across a variety of Head Start and other early childhood programs before policymakers can understand the practical issues involved in implementing the tools at scale and the usefulness of the information they provide, and establish baseline data on current norms and variations on these measures across the Head Start population.
Additional research is also needed in order to develop valid and reliable measurement tools in domains where they are lacking. For example, while we know that family engagement and stability are crucial to children’s early development and future school success, there is little evidence about which measurable parent and family outcomes are most important for Head Start programs to measure (see box 4). Similarly, there are few valid and reliable tools measuring children’s social-emotional development.
Even where essential indicators seem easy to measure, it is not always clear how to best collect or present data to provide useful information about program performance. Child attendance offers a case in point. Measuring attendance seems like a no-brainer — either a child is there or she isn’t — and Head Start programs already track data on attendance. But the way attendance data are collected and shared directly affects how they can be used to address key questions: Which children are absent the most? Is there a pattern to their absenteeism? How do interventions or changes in practice affect children’s attendance? Similar questions arise about many potential indicators of safety, health, and financial quality. If a performance measurement system is to include information on these indicators, OHS will need to establish common standards for how programs collect and report data, in ways that maximize the data’s value to programs and other users.
These challenges are real — but they cannot become an excuse for maintaining the status quo. Rather, what’s needed is a strong commitment to moving toward more results-based measures of Head Start performance, coupled with an iterative and thoughtful process for developing and refining those performance measures and evaluation methods over time.
MONEYBALL FOR HEAD START 19
Box 4: Developing Measures of Parent, Family, and Community Engagement Study
Parent engagement and leadership is a core principle of Head Start, but efforts to measure program results have historically focused more on measuring child development and learning and less on family growth and engagement. Programs lack appropriate tools for gathering quantitative data about family engagement and must rely on measures of outputs, such as parent attendance at events, rather than capture meaningful changes in family outcomes over time.
To address that issue, in spring 2015, the National Head Start Association partnered with NORC — a research institute at the University of Chicago — and the Region V Head Start Association to launch a new study of Head Start parent, family, and community engagement. Assessing Parent, Family, and Community Engagement through Parent Report will:
» inform the national understanding of parent engagement; and
» create a tool to help local programs collect better information about parents’ experiences in Head Start.
During the first phase of the project, in fall 2015, the study gathered stories from parents of children who participated in a random sample of Head Start, Early Head Start, Migrant and Seasonal Head Start, and American Indian and Alaska Native Head Start programs. The team at NORC will use computational text analysis to identify the themes parents expressed as being the most valuable Head Start contributions to their children and families. The analysis will also explore how these themes align with the Head Start Parent, Family, and Community Engagement Framework.24 Ultimately, the study will produce a measurement tool that programs across the country can easily use to measure parent, family, and community engagement based on parent stories, focusing on the very dimensions Head Start families have identified as most important. The tool is intended to enable programs to measure individual families’ changes in engagement over time, support measurement of program changes from year to year, and enable comparisons among Head Start grantees. It will provide programs with useful information to both enhance engagement with families and support ongoing program evaluation activities.
— Emmalie Dropkin, National Head Start Association