All data collected through the evaluation were managed and stored by Dr. Elm, the external evaluator. The computer used for storage and analysis was located in a locked office. Only the external evaluator had access to the raw data. Data were backed up weekly to an external drive, which was kept in a locked drawer. To protect teacher and student privacy, identification numbers were assigned to all participants. Teacher and student names were not recorded with the data.
READ online records regarding student and teacher use, rubric data, and survey data were only accessible by the external evaluator. Results that were released were only in aggregation and had no identifying information. All evaluation data were secured and kept confidential to protect individual privacy.
Step 4: Interpret the Results The READ evaluation subcommittee (the E-Team) examined the evaluation results. Some highlights from the findings are provided below. Use of these findings will be discussed in Step 5: Inform and Refine – Using the Results.
Summative
1. First-year results indicated that READ and state reading scores for READ students were higher than those for non-READ students. The gains were especially compelling for students in classrooms in which READ was used regularly and with fidelity, where increases in reading scores were over three times that of non-READ students.
2. Students in classrooms where READ was used regularly and with fidelity increased their reading scores by twice that of students in READ classrooms where READ was used minimally.
3. Students of teachers who used READ assessment data as intended to differentiate instruction increased their reading scores on the state assessment by twice as much as students of teachers who did not use READ assessment data as intended.
4. Student scores on READ assessments had a statistically significant and strong positive correlation with student scores on the state reading assessment, indicating that these two assessments are likely well aligned and that READ assessment data are likely a good indicator of performance on the state reading assessment.
Formative
5. Student assessment data could not be analyzed by home use of the READ program because only one classroom implemented the home component.
6. At the start of the year, teacher use of READ was promising, and the program met its targets. However, as the program progressed and as more teachers were pressed to
107
improve their use of READ, several targets were not met. READ student assessment data were not used as regularly by teachers as the classroom component of READ.
A full accounting of evaluation results by logic model component and evaluation question is provided in tables 13, 14, and 15.
108
Table 13: READ Evaluation Results—Strategies and Activities/Initial Implementation
Logic Model Components Evaluation Questions
Indicators Targets Evaluation Findings
Interactive, standards- based classroom lessons (using the READ software with interactive classroom technologies and individual handheld mobile devices for each student)
To what extent did teachers have access to the necessary technology in the classroom to use READ in their instruction?
Increased number of teachers with access to the necessary technology in their classroom to use READ
By the start of the school year, all teachers will have the necessary technology in their classroom to use READ.
By September, all READ teachers (100%) had the necessary technology in their classroom to use READ.
Standards-based reading assessments (Internet- based, formative assessments of student reading skills administered within the READ software)
To what extent were READ assessments made available to students and teachers?
Increased number of teachers with access to READ assessments
Increased number of students with access to READ assessments
By the start of the school year, all teacher accounts will have been set up in READ.
By the end of September, all student accounts will have been set up in READ.
By September, all READ teacher accounts (100%) had been set up.
By the end of September, all student accounts (100%) had been set up.
109
Logic Model Components Evaluation Questions
Indicators Targets Evaluation Findings
Standards-based reading homework (Internet- based using READ software)
To what extent did students have access to READ at home?
Increased number of students with access to READ at home
By the end of September, all teachers will have determined how many students have the technology necessary to access READ from home.
By the end of September, all teachers (100%) had determined how many students had home access to READ.
→ At three of the schools, most students (90%) had home access to READ.
→ At two of the schools, about half the students (54%) had home access to READ.
→ At one school, less than 20% of students had the technology necessary to access READ from home.
→ Even within those schools that had high numbers of students with home access, the classroom variability was large. Only one of the 40 classrooms had 100% of students with access to READ from home.
Teacher professional development on integrating READ into classroom instruction (using an interactive wireless pad)
To what extent did teachers receive professional development on how to integrate READ into their classroom instruction?
Increased number of teachers trained in how to effectively use READ in their classroom instruction
By the start of the school year, all teachers will have received professional development on how to integrate READ into their classroom instruction.
By September, all teachers (100%) had received professional development on how to integrate READ into their classroom instruction.
110
Logic Model Components Evaluation Questions
Indicators Targets Evaluation Findings
Teacher professional development on using READ assessment data for classroom lesson planning
To what extent did teachers receive professional development on how to incorporate READ assessment data into their classroom lesson planning?
Increased number of teachers trained in how to use READ assessment data in their lesson planning
By the start of the school year, all teachers will have received professional development on how to use READ assessment data in their lesson planning.
By September, all teachers (100%) had received professional development on how to use READ assessment data in their classroom lesson planning.
Student training on using READ (in the classroom and at home)
To what extent were students trained in how to use READ?
Increased number of students trained in how to use READ
By the end of September, all teachers will have trained their students in the use of READ (for use in the classroom and at home).
By October, all teachers (100%) had trained their students in the classroom use of READ.
By the end of October, only one teacher (2%) had trained his students in the home use of READ.
→ Open-ended survey items revealed that teachers chose not to train students in the home use of READ unless every student in the classroom was able to take advantage of the home component. Since only one classroom had 100% participation, only one teacher trained students on home use.
111
Table 14: READ Evaluation Results—Early/Short-Term and Intermediate Objectives
Logic Model Components Evaluation Questions
Indicators Targets Evaluation Findings
Increased student use of READ at home (early/short-term)
How often did students receive READ homework assignments?
Increased number of teachers assigning READ homework
By November, over 50% of teachers will be assigning weekly READ homework.
By November, only one teacher (2%) was assigning weekly READ homework.
Increase d student use of READ at home (early/ short -term)
To what extent did students complete READ homework assignments? **Note frequency and duration of use.
Increased number of students completing READ homework within a reasonable time
By December, over 50% of students will be completing weekly READ homework.
Students will spend no more than 20 minutes to complete weekly READ homework. (Note: Completion rates and duration of use are available through the READ online system.)
By December, most students (70%) in the classroom where the READ homework component was used were completing the assignment. These students spent, on average, 15 minutes to complete the weekly READ homework.
