occur early on through READ homework assignments and teacher use of READ in the classroom, both of which the E-Team identified as indicators of exposure. Then at the intermediate stage, student interaction during classroom lessons using READ (measured by the rubric described below) was assumed to further increase student exposure.
The evaluation matrix is presented in three tables:
1. Strategies and Activities (Table 7: Evaluation Matrix Addressing Strategies and Activities During the Initial Implementation—Indicators and Targets)
2. Early/Short-Term and Intermediate Objectives (Table 8: Evaluation Matrix Addressing Early/Short-Term and Intermediate Objectives—Indicators and Targets)
3. Long-term Goals (Table 9: Evaluation Matrix Addressing Long-Term Goals—Indicators and Targets)
As you read through the tables, you will see that the evaluation will collect much of the data through the four instruments described below: (1) the READ implementation rubric, (2) the teacher survey, (3) the READ student assessment, and (4) the annual state student reading assessment.
1. READ implementation rubric: The E-Team created the READ implementation rubric to examine the quality of teacher practice when using READ during classroom instruction, student interaction during learning, teacher integration of READ into classroom instruction, and teacher use of READ student assessment data to plan and differentiate instruction. Dr. Elm will administer the READ implementation rubric on a monthly basis, alternating between classroom observations one month and interviews with teachers the following month.
2. Teacher survey: Dr. Elm will conduct the teacher survey in October as a baseline and again in December, February, April, and June. The teacher survey has multiple sections including some open-ended questions. Most items will be included every time the survey is administered. Others (such as items on the initial account setup and access) will be administered only when appropriate.
3. READ student assessment: The READ software itself includes an embedded, formative, standards-based READ assessment to measure student learning before and after each lesson. The data from the embedded READ assessment are stored in the READ system for teachers to use in assessing student learning and planning their instruction. The evaluation also will use these data.
4. State student reading assessment: The evaluation also will use state reading assessment scores to measure student learning in reading. The state reading assessment is administered in April of each academic year. Reading scores from the spring prior to READ program implementation will be used as a baseline against which to measure student reading improvement.
84
The evaluation will use data collected using each of these four instruments—the READ implementation rubric, teacher survey, READ student assessment, and reading scores from the state assessment. READ student assessment data and state student reading assessment data will be disaggregated and examined by quality of teacher use (using the READ implementation rubric), frequency of home use, initial reading performance on state assessments, grade level, gender, ethnicity, special education status, and English language proficiency. In addition, reading scores on the READ student assessment will be analyzed in relation to the state assessment reading scores to determine the degree to which the READ assessments correlate with the state reading assessment.
Tables 7, 8, and 9 show the evaluation questions, indicators, and targets developed in Step 2: Plan the Evaluation.
Table 7: Evaluation Matrix Addressing Strategies and Activities During the Initial Implementation—Indicators and Targets
Logic Model Components Evaluation Questions Indicators Targets
Interactive, standards- based classroom lessons (using the READ software with interactive classroom technologies and individual handheld mobile devices for each student)
To what extent did teachers have access to the necessary technology in the classroom to use READ in their instruction?
Increased number of teachers with access to the necessary technology in their classroom to use READ
By the start of the school year, all teachers will have the necessary technology in their classroom to use READ.
Standards-based reading assessments (Internet- based, formative assessments of student reading skills administered within the READ software)
To what extent were READ assessments made available to students and teachers? Examine overall, by school, and by grade level.
Increased number of teachers with access to READ assessments
Increased number of students with access to READ assessments
By the start of the school year, all teacher accounts will have been set up in READ.
By the end of September, all student accounts will have been set up in READ.
Standards-based reading homework (Internet-based using READ software)
To what extent did students have access to READ at home? Examine overall and by grade level, race, gender, and socioeconomic status.
Increased number of students with access to READ at home
By the end of September, all teachers will have determined how many students have the technology necessary to access READ from home.
85
Logic Model Components Evaluation Questions Indicators Targets
Teacher professional development on integrating READ into classroom instruction (using an interactive wireless pad)
To what extent did teachers receive professional development on how to integrate READ into their classroom instruction?
Increased number of teachers trained in how to effectively use READ in their classroom instruction
By the start of the school year, all teachers will have received professional development on how to integrate READ into their classroom instruction.
Teacher professional development on using READ assessment data for classroom lesson planning
To what extent did teachers receive professional development on how to incorporate READ assessment data into their classroom lesson planning?
Increased number of teachers trained in how to use READ assessment data in their lesson planning
By the start of the school year, all teachers will have received professional development on how to use READ assessment data in their lesson planning.
Student training on using READ (in the classroom and at home)
To what extent were students trained in how to use READ?
Increased number of students trained in how to use READ
By the end of September, all teachers will have trained their students in the use of READ (for use in the classroom and at home).
Table 8: Evaluation Matrix Addressing Early/Short-Term and Intermediate Objectives— Indicators and Targets
Logic Model Components Evaluation Questions
Indicators Targets
Increased student use of READ at home (early/short- term)
How often did students receive READ homework assignments?
To what extent did students complete READ homework assignments? **Note frequency and duration of use.
Increased number of teachers assigning READ homework
Increased number of students completing READ homework, within a reasonable time
By November, over 50% of teachers will be assigning weekly READ homework.
By December, over 50% of students will be completing weekly READ homework assignments. Students will spend no more than 20 minutes to complete READ homework. (Note: Completion rates and duration of use are available through the READ online system.)
86
Logic Model Components Evaluation Questions
Indicators Targets
Increased teacher use of READ in the classroom (early/short-term)
In what ways and how often did teachers use READ in the classroom with students? **Note frequency, duration, and nature of use.
Increased number of teachers using READ in the classroom with students
Improved teacher use of READ in the classroom with students
By October, all teachers will be using READ in the classroom with students.
By November, 25% of teachers will score a 2 or above (out of 4) on the READ implementation rubric.
By December, 50% of teachers will score a 2 or above on the READ implementation rubric.
Increased student exposure to standards- based learning opportunities (early/short- term)
To what extent did students complete READ homework assignments?
Increased number of students completing READ homework
By December, over 50% of students will be completing weekly READ homework assignments.
Increase d student exposure to standards-based learning opportunities (early/short-term)
How often did teachers use READ in the classroom with students?
Increased number of teachers using READ in the classroom with students
By October, all teachers will be using READ in the classroom with students.
Increased availability of standards-based, formative READ assessment data on student reading performance (early/short- term)
How often did teachers access READ student assessment data? **Note frequency and type of access.
Increased number of teachers accessing READ student assessment data
By October, 50% of teachers will have accessed READ student assessment data.
By November, all teachers will have accessed READ student assessment data.
Increased teacher use of standards-based READ assessment data (early/short-term)
In what ways did teachers use READ student assessment data?
Improved teacher use of READ student assessment data
By February, 25% of teachers will score a 3 or above on the READ implementation rubric.
Increased student interaction during learning (intermediate)
To what extent and how did students interact during classroom instruction when READ was used? **Note frequency and type of interaction.
