To what extent did student learning improve after READ was implemented?
To what extent did learning outcomes vary with teacher use of READ in the classroom?
To what extent did learning outcomes vary with teacher use of READ assessment data to plan and differentiate instruction?
How did student performance on the READ assessments correlate with student performance on state assessments?
In what ways did learning outcomes vary by initial reading performance on state assessments?
In what ways did learning outcomes vary by grade level?
In what ways did learning outcomes vary by special education status and English language proficiency?
In what ways did learning outcomes vary with the frequency of READ use at home?
29
If you do not have the resources to focus on all of your evaluation questions, you may need to prioritize. When prioritizing evaluation questions, it is important to have at least some measurement in all three categories: implementation, short-term/intermediate objectives, and long-term goals.
What Data Should I Collect? Now that you have developed your logic model and decided on your evaluation questions, the next task is to plan how you will answer those questions. Your logic model is your road map during this process. Just as you used the key components of your logic model as a guide to develop your evaluation questions, your evaluation questions will drive the data that will be collected through your evaluation.
The answers to your evaluation questions will give you the information you need to know in order to improve your program and to make critical program decisions. The following paragraphs will take you through the process of creating indicators for your evaluation questions that relate to program strategies and activities, short-term objectives, intermediate objectives, and long-term goals. Your indicators will dictate what data you should collect to answer your evaluation questions.
Indicators are statements that can be used to gauge progress toward program goals and objectives. An indicator is a guide that lets you know if you are moving in the right direction. Your indicators will be derived from your evaluation questions; for some evaluation questions, you might have multiple indicators. Indicators are the metrics that will be tied to targets or benchmarks, against which to measure the performance of your program.
Indicators can be derived from evaluation questions and are used to measure progress toward program goals and objectives. An evaluation question may have one or more indicators.
Targets provide a realistic time line and yardstick for your indicators. Indicators and targets should have the following characteristics:
An indicator is SMA:
Specific
Measurable
Agreed upon
And a target is RT:
Realistic
Time-bound
Together, indicators and targets are SMART.
30
Indicators and targets should be specific, measurable, agreed upon, realistic, and time-bound (SMART). For instance, suppose you are evaluating a teacher recruitment and retention program. You may have an
objective on your logic model that states “to increase the number of highly qualified teachers in our school district” and a corresponding evaluation question that asks “to what extent was the number of highly qualified teachers increased in our school district?” However, we know there are several ways that a “highly qualified teacher” can be defined, such as by certifications, education, content knowledge, etc. The indicator would specify the definition(s) that the evaluator chooses to use and the data element(s) that will be collected. For example, to be specific and measurable, the indicator might be twofold: “increasing number and percentage of teachers who are state certified” and “increasing number and percentage of teachers who hold National Board certification.” At this point, it would also be wise to consider whether the indicators you choose are not only measurable, but also available, as well as agreed upon by the evaluation team and program staff.
Next, clarify your indicator by agreeing upon a realistic and time-bound target. Thus, a target is a clarification of an indicator. A target provides a yardstick and time line for your indicator, specifying how much progress should be made and by when in order to determine to what extent goals and objectives have been met. Targets for the above example might include: “by 2015, all teachers in the school district will be state certified” and “by 2018, 50 percent of district teachers will have National Board certification.” For some programs, it is possible that reasonable targets cannot be set prior to the program’s operation. For instance, consider a program that is intended to improve writing skills for seventh graders, and the chosen indicator is a student’s score on a particular writing assessment. However, the evaluation team would like to see baseline scores for students prior to setting their target. In this case, a pretest may be given at the start of the program and, once baseline scores are known, targets can be determined.
31
Table 1
With the evaluation questions that the READ oversight team and E-Team had created, the E-Team was ready to expand on each with indicators and accompanying targets. Using the logic model as its guide, the E-Team created an evaluation matrix detailing the logic model component, associated evaluation questions, indicators, and accompanying targets. Two examples are provided below. All indicators for the READ project are provided in Appendix A.
1. To what extent were READ assessments made available to students and teachers? (activity) Indicator: Increased number of students and teachers with access to READ assessments. Targets: By the start of the school year, all teacher accounts will have been set up in READ. By the end of September, all student accounts will have been set up in READ.
2. In what ways and to what extent did teachers integrate READ into their classroom instruction? (intermediate objective) Indicator: Improved integration of READ lessons into classroom instruction, as measured by teacher scores on the READ implementation rubric (rubric completed through classroom observations and teacher interviews). Targets: By April, 50% of teachers will score a 3 or above (out of 4) on the READ implementation rubric. By June, 75% of teachers will score a 3 or above and 25% of teachers will score a 4 on the READ implementation rubric.
32
Evaluation Matrix Now that you have created evaluation questions with accompanying indicators and targets for each component of your logic model, how do you organize that information into a usable format for your evaluation? One method is to use an evaluation matrix. An evaluation matrix represents your logic model components, evaluation questions, indicators, and targets by your logic model strategies and activities, early and intermediate objectives, and long-term goals.
shows an example shell. Table 26: Evaluation Matrix Template is provided in Appendix E. Information for completing the data source, data collection, and data analysis columns will be covered next in the guide.
Table 1: Evaluation Matrix Example Shell
Lo gi
c M
od el
Co
m po
ne nt
Ev al
ua tio
n Q
ue st
io ns
In di
ca to
rs
Ta rg
et s
Da ta
S ou
rc es
Da ta
C ol
le ct
io n
Da ta
A na
ly si
s
