Increased student interaction during learning, as measured by the student component of the READ implementation rubric (rubric completed through classroom observations and teacher interviews)
By February, 25% of classrooms will have a score of 3 or above (out of 4) on the READ implementation rubric.
By April, 50% of teachers will score a 3 or above on the READ implementation rubric.
87
Logic Model Components Evaluation Questions
Indicators Targets
Improved integration of READ into classroom instruction (intermediate)
In what ways and to what extent did teachers integrate READ into their classroom instruction? **Note the quality with which READ was integrated into classroom instruction by teachers.
Improved integration of READ lessons into classroom instruction, as measured by teacher scores on the READ implementation rubric (rubric completed through classroom observations and teacher interviews)
By April, 50% of teachers will score a 3 or above on the READ implementation rubric.
By June, 75% of teachers will score a 3 or above and 25% of teachers will score a 4 on the READ implementation rubric.
Improved differentiation of instruction (intermediate)
In what ways and to what extent did teachers use READ assessment data to plan and differentiate instruction? **Note what data were used and how data were used in instructional planning.
Increased number of teachers using READ assessment data to plan instruction
Improved use of READ assessment data to differentiate instruction
By December, all teachers will be using READ assessment data on a weekly basis to plan instruction.
By April, 50% of teachers will score a 3 or above on the READ implementation rubric.
By June, 75% of teachers will score a 3 or above and 25% of teachers will score a 4 on the READ implementation rubric.
88
Table 9: Evaluation Matrix Addressing Long-Term Goals—Indicators and Targets
Logic Model Components
Evaluation Questions Indicators Targets
Increased student engagement in reading
In what ways did READ foster student engagement during reading lessons?
Increased frequency and improved quality of student engagement in the classroom, as measured by the READ implementation rubric
By February, 25% of classrooms will have a score of 3 or above (out of 4) on the READ implementation rubric.
By April, 50% of classrooms will score a 3 or above on the READ implementation rubric.
By June, 75% of classrooms will score a 3 or above and 25% of teachers will score a 4 on the READ implementation rubric.
89
Logic Model Components
Evaluation Questions Indicators Targets
Improved student reading skills
To what extent did READ improve student learning in reading?
→
→
→
→
→
→
→
→
To what extent did student learning improve after READ was implemented?
To what extent did learning outcomes vary with teacher use of READ in the classroom?
To what extent did learning outcomes vary with teacher use of READ assessment data to plan and differentiate instruction?
How did student performance on the READ assessments correlate with student performance on state assessments?
In what ways did learning outcomes vary by initial reading performance on state assessments?
In what ways did learning outcomes vary by grade level?
In what ways did learning outcomes vary by special education status and English language proficiency?
In what ways did learning outcomes vary with the frequency of READ use at home?
Increased scores on tests assessing students’ reading ability (including both state assessments and the formative assessments provided within the READ software)
Within 2 years, the increase in student scores on the state standards-based reading assessment will be statistically significant for those students who participated in READ versus those students who did not participate in READ.
State reading scores and READ assessment data will be disaggregated and examined by quality of READ teacher use (using the READ implementation rubric), frequency of READ home use, initial reading performance on state assessments, grade level, gender, ethnicity, special education status, and English language proficiency.
Reading scores on the state assessment will be analyzed in relation to scores on the READ assessment data, in order to determine the degree to which READ assessments correlate with the state assessments.
Evaluation Design Grovemont School District had 80 third- through fifth-grade classrooms across six elementary schools (28 third-grade classrooms, 28 fourth-grade classrooms, and 24 fifth-grade classrooms). District class size for grades 3 through 5 ranged from 22 to 25 students per classroom. Because of state budget cuts and reduced funding for the program, the E-Team knew that Mrs. Anderson and the READ oversight team would have to make some difficult choices about how to structure and evaluate their program.
90
Some members of the oversight team wanted to implement the program in fifth grade only for the first year, and then reexamine funds to see if they might be able to expand down to fourth grade in Year 2. Others voted to start the program at two of the six elementary schools and then try to include an additional school in Year 2. Dr. Elm and the E-Team recommended that they consider partially implementing the program at all six schools and across all three grades.
Dr. Elm explained that they would receive much better information about how their program was working and, more importantly, how it could be improved, if they were able to compare results from those classrooms that were using the program with those that were not. Dr. Elm knew that students at all of the schools in Grovemont School District were randomly assigned to teachers during the summer before each school year. However, Dr. Elm explained that in order to minimize initial differences between those classrooms that participate in READ and those that do not, they should consider randomly assigning half of the classrooms to continue with the existing district curriculum while the other half would supplement their existing curriculum with the READ program. Dr. Elm also recommended that they first divide the classrooms by school and grade level so that each school and grade would have one half of the classrooms assigned to the program. Teachers whose classrooms were not assigned to the program would be assured that if the program proved successful, they would be on board by Year 3. However, if the program did not have sufficient benefits for the students, it would be discontinued in all classrooms after Year 2. Dr. Elm concluded that building a strong evaluation into their program would provide them with credible information as to how their program was working and that having data to direct their program adjustments and improvements would give the program the best opportunity to be successful. The READ oversight team agreed to think about this idea and reconvene in 1 week to make a decision.
The E-Team also distributed the evaluation matrix it had created based on the READ logic model. The E-Team asked the oversight team to review the matrix and provide any feedback or comments.
The following week, the E-Team and READ oversight team reconvened to decide how to structure the program and to work on the evaluation design. Mrs. Anderson had spoken with the district superintendent about the evaluator’s suggestion of implementing READ in half the district’s third- through fifth-grade classrooms, with the promise that it would be expanded to all classrooms in Year 3 if the program was successful. Although logistically it would be easier to implement the program in two or three schools or one or two grades than to implement it in half the classrooms in all schools and at all grades, the superintendent understood the benefit of the added effort. The evaluation would provide higher quality data to inform decisions for program improvement and decisions regarding the program’s future.
Mrs. Anderson shared the superintendent’s comments with the oversight team and evaluation subcommittee. Like the superintendent, team members felt conflicted by the choice between
91
simpler logistics or a stronger evaluation design. Dr. Elm understood the dilemma all too well, but as an evaluator and an educator, she believed that a strong evaluation would result in improved program implementation and improved program outcomes.
Dr. Elm recognized that implementing the program in all classrooms in one grade level across the district would offer the weakest evaluation design and the least useful information but would likely be the simplest option logistically. Another option would be to start the program in all classrooms at two or three schools. In such a case, the other schools could be used as comparisons. For this reason, Dr. Elm explored the comparability of the six elementary schools in case the team decided to go that route. Five of the elementary schools had somewhat comparable state test scores in reading, while the sixth school had lower state test scores, and the difference was statistically significant. In addition, Schools 1 through 5 had similar (and fairly homogenous) populations, while School 6 had a much lower socioeconomic student population and a much higher percentage of ELL students. Because the district was interested in how the program worked with ELL students, the team knew that the evaluation needed to include School 6. However, if School 6 were used in a three-school implementation, the team would not have a comparable school against which to benchmark its results.
While not the simplest option, the oversight team decided that its best option would be to structure the program in such a way as to maximize the quality of the information from the evaluation. The team chose to build a strong evaluation into the READ program design to provide the formative information needed for program improvement and valid summative information for accountability.
