Based on the READ oversight team’s decision about how to structure the program, Dr. Elm and the E-Team drafted the following evaluation design. They presented the design at the next oversight team meeting. The oversight team voted to approve the design as follows:
Design: Multiple-group, experimental design (students randomly assigned to classrooms by the school prior to the start of the school year and classrooms randomly assigned to the READ program group or a non-READ comparison group).
Program group (READ): 40 classrooms (22 to 25 students per classroom).
Comparison group (non-READ): 40 classrooms (22 to 25 students per classroom).
Classrooms will be stratified by grade level within a school and randomly assigned to either the READ program group or a comparison group. The READ and non-READ groups will each include 14 third-grade classrooms, 14 fourth-grade classrooms, and 12 fifth- grade classrooms.
92
Enriching the evaluation design: Program theory and logic modeling will be used to examine program implementation as well as short-term, intermediate, and long-term outcomes.
Data Collection Methods The E-Team decided on data collection methods, including the data sources, for each evaluation question and associated indicators. Two examples are provided below.
1. In what ways and to what extent did teachers integrate READ into their classroom instruction?
→
→
→
→
A READ rubric will be used to measure teacher implementation of READ in the classroom.
The rubric will be completed through classroom observations and teacher interviews.
The READ implementation rubric will be on a 4-point scale, with a 4 representing the best implementation.
Data will be collected monthly, alternating between classroom observations one month and interviews the following month.
2. To what extent did READ improve student learning in reading?
→
→
→
→
→
The state reading assessment will be used to measure student learning in reading. It is administered in April of each academic year, beginning in second grade.
READ assessment data will be used as a formative measure to examine student reading performance.
State reading scores and READ assessment data will be disaggregated and examined by quality of teacher use (using the READ implementation rubric), frequency of home use, initial reading performance, grade level, gender, ethnicity, special education status, and English language proficiency.
Previous year state reading assessment scores will be used as a baseline against which to measure student reading improvement.
Reading scores on the state assessment will be analyzed in relation to scores on the READ assessments in order to determine the degree to which READ assessments correlate with the state reading assessment.
For a full list of evaluation questions, data sources, and data collection methods, see the READ Evaluation Matrix tables 10, 11, and 12 in Step 3.
93
Step 3: Implement the Evaluation The READ external evaluator collected a mix of quantitative and qualitative data to address evaluation questions. Qualitative data collected through observations and interviews were coded using the READ implementation rubric and analyzed using descriptive statistics, including means and frequency distributions. Student reading assessment data were analyzed by testing for statistical significance, comparing mean test scores between groups of students and over time.
The following is an example using one of the READ intermediate objectives:
1. Logic Model Component: Improved integration of READ into classroom instruction (intermediate objective).
2. Evaluation Question: In what ways and to what extent did teachers integrate READ into their classroom instruction?
3. Indicator: Improved integration of READ lessons into classroom instruction.
4. Targets: By April, 50% of teachers will score a 3 or above (out of 4) on the READ implementation rubric. By June, 75% of teachers will score a 3 or above on the READ implementation rubric.
5. Data Source: READ implementation rubric (developed by the E-Team and administered by Dr. Elm)
6. Data Collection: Rubric completed through alternating, monthly classroom observations and teacher interviews.
7. Data Analysis: Rubric scores aggregated into frequency distributions and means; change over time to be analyzed.
The full READ Evaluation Matrix is included in tables 10, 11, and 12. Note that the evaluation matrix was completed in steps. The logic model components are taken directly from the READ logic model created in Step 1: Define the Program. The logic model components consist of strategies and activities, early/short-term and intermediate objectives, and long-term goals. The evaluation questions were created in Step 2: Plan the Evaluation, guided by the READ logic model. Indicators and targets were derived in Step 2 using the READ logic model and evaluation questions. At the end of Step 2, data collection sources and methods were chosen for each READ indicator. Data analysis methods were determined in Step 3: Implement the Evaluation. (See Appendix E for Table 26: Evaluation Matrix Template.)
94
Table 10: READ Evaluation Matrix—Strategies and Activities/Initial Implementation
Logic Model Components
Evaluation Questions
Indicators Targets Data Sources Data Collection Data Analysis
Interactive, standards-based classroom lessons (using the READ software with interactive classroom technologies and individual hand- held mobile devices for each student)
To what extent did teachers have access to the necessary technology in the classroom to use READ in their instruction?
Increased number of teachers with access to the necessary technology in their classroom to use READ
By the start of the school year, all teachers will have the necessary technology in their classroom to use READ.
Technology installation records; teacher survey
Technology installation records examined in September for evidence of necessary classroom technology
Teacher survey administered in October, including items on technology in the classroom
Records analyzed with basic descriptive statistics (counts and percentages) of classrooms with necessary technology
Teacher survey analyzed with basic descriptive statistics including means and frequency distributions; open- ended items on the survey summarized, and if warranted, analyzed for themes
95
Logic Model Components
Evaluation Questions
Indicators Targets Data Sources Data Collection Data Analysis
Standards-based reading assessments (Internet-based, formative assessments of student reading skills administered within the READ software)
