Appendix C: Evaluation Resources and Appendix D: Evaluation Instruments for Educational Technology Initiatives include evaluation resources and information about instruments that you may find useful for your evaluations. Appendix E: Evaluation Templates includes a logic model template you can use to define your program and an evaluation matrix template to use to plan your evaluation. Finally, Appendix F: Lists of Tables and Figures appears at the end of the guide.
Figure 1: Embedded Evaluation Model This figure illustrates the five-step, iterative evaluation process: define, plan, implement, interpret, and inform and refine.
Step 1 involves defining the program and logic. Ask these questions about the program: What is the program? What does the program purport to accomplish? What are the goals and objectives? What are the strategies and activities?
Ask these questions about the logic: How do program strategies related to program goals? What is the underlying logic of the program? What are the program’s short-term, intermediate, and long-term objectives? To what extent is program theory supported by rigorous research?
Step 2 involves planning the design. Ask these questions about the design: What questions should the evaluation answer? What indicators best address objectives? What evaluation methods should be used? What is the strongest design that can be feasibly implemented?
Step 3 involves implementation. Ask these questions about the evaluation: How should data be collected? How should data be organized and maintained? How should data be analyzed to best answer evaluation questions?
Step 4 involves interpreting the results. Ask these questions about the results: How should results be interpreted? How can the program be improved? To what extent did the program accomplish its goals? How should results be communicated? What can be done to make sure that evaluation results are used?
Step 5 has two parts: 5a is inform, and 5b is refine.
8
Embedding Evaluation Into the Program STEP 1: DEFINE – What Is the Program?
How Can I Find Out More About the Program? (Understanding the Program) The first step to conducting your evaluation is to understand what you want to evaluate. Whether you are evaluating a new program or a program that you have been using for some
For the past 5 years, reading scores in the Grovemont School District have been declining. The curriculum supervisor, Mrs. Anderson, has tried many strategies to improve reading skills. However, scores continue to decline. Mrs. Anderson has been searching for curricular and assessment materials that are better aligned with state reading standards and that provide ongoing standards-based assessment data. Mrs. Anderson found a program called READ (Reading Engagement for Achievement and Differentiation) that looked promising. After reviewing research on the program and documentation from the vendor as well as numerous discussions and interviews with other districts that had implemented the program, Mrs. Anderson and the district superintendent decided to present the READ program to the school board, in order to gain approval for funding the program for Grades 3-5.
At last month’s meeting, the school board voted to partially fund the READ program. Due to recent state budget cuts, the school board was only able to fund the program at 50% for 2 years. At the end of the 2 years, the board agreed to revisit its funding decision. The board required an evaluation report and presentation due in September of each year.
Before starting to plan the READ program, Mrs. Anderson invited one teacher from each of the district’s six elementary schools, the district reading coach, one of the district’s reading specialists, and the district technology coordinator to join the READ oversight team. This 10-member team was charged with planning the READ program and its evaluation. The team asked an evaluator from the local university to conduct the READ evaluation and to attend oversight team meetings.
Note: The examples set out in this guide are provided solely for the purpose of illustrating how the principles in this guide can be applied in actual situations. The programs, characters, schools, and school districts mentioned in the examples are fictitious.
9
time, it is still important to begin from the basics in understanding how a program works. Do not rely on what you already know about the program or what you believe the program is intended to accomplish. Instead, take what you know, and build upon it with information from multiple sources. By doing this, you will have a full understanding of the program including multiple perspectives and expectations, as well as basic underpinnings and complex inner workings.
So, how do you find out more about the program? If you have experience with the program, you should first document what you know. You may want to investigate if any rigorous previous evaluations have been conducted of the program. If well designed and well carried out, previous evaluations can provide useful information regarding how a program operates.
Another good source from which you can learn more about the program is existing documentation. Documents such as technology plans, curriculum materials, strategic plans, district report cards, user manuals, and national, state, or district standards may have useful information for understanding your program and the context in which it will be implemented. Further, you may want to talk with people who are most familiar with the program, such as vendors and people from other districts that have implemented the program. Consider
The oversight team asked the external evaluator, Dr. Elm, to help them plan the evaluation. Dr. Elm suggested that the oversight team build evaluation into its program as the team is designing it. By embedding evaluation into the program, information from the evaluation would be available to guide program implementation. Evaluation data would both drive program improvement and be the foundation for future decisions regarding whether the program should be continued, expanded, scaled down, or discontinued.
The oversight team members invited Dr. Elm to lead them through the process of building evaluation into their program planning. Dr. Elm explained that the first step is to gain a thorough understanding of the program. In doing this, Mrs. Anderson shared the materials she had already reviewed with the oversight team. In addition, the oversight team contacted four school districts that had used the READ program successfully in order to learn more about the program. To develop a thorough and shared understanding of the context in which the READ program would be implemented, the team reviewed the state’s reading standards, the district’s strategic plan, the district’s core learning goals and curriculum maps in reading, and the district’s technology plan. The team also examined reading grades and state reading assessment scores for the district as a whole, as well as by school, English Language Learner (ELL) status, and special education status for the past 5 years.
10
conducting interviews and group discussions to learn more about their insight into the program, how it operates, and what goals it is intended to achieve.
Why Should the Program Work? (Explaining the Program Theory) Once you have a good understanding of the program, the next step is for you to document more thoroughly what you know about the program. The first component in explaining the program is to describe the program’s goals and objectives. Goals should reflect a shared understanding among program stakeholders as to what the program should achieve. What is the program intended to accomplish? How would you know if it worked? If the program were a success, what would have happened? What would have changed?
The next step, stated Dr. Elm, is to define the program by explaining the program theory. Explaining the program theory will include what the program is intended to accomplish, as well as how and why the program is expected to work. Dr. Elm recommended that the team complete the program theory in three parts: (a) defining the program’s long-term goals, (b) delineating the program’s strategies and activities, and (c) explaining how and why the team believes the program’s activities and strategies will result in the desired outcomes.
Your program may have one or two goals, or your program may have many goals. For some programs, the primary goal may be to improve student learning. For others, primary goals might be to affect teacher content knowledge and teacher practice. Goals may have to do with behavior, safety, involvement, or attitudes. The first piece in explaining the program is to list the overall goals of your program or initiative. Goal statements should be broad and general and should reflect the overall intent of your program or a shared vision of what your program is supposed to accomplish. Objectives tend to be more specific and are often short term or intermediate term. If objectives are known, record them. However, at this point in program planning, broad goal statements are sufficient.
Based on their review of documentation and research as well as discussions and interviews with other districts that have implemented the program, and from meetings with district administration and school staff, Mrs. Anderson and the oversight team set the following long-term goals for READ:
1. Increased student engagement in reading
2. Improved student reading skills
11
Once you have documented what the program is intended to accomplish, the next component is to document your program’s strategies and activities. How will the program accomplish these goals? What strategies will be used to achieve your goals? What activities will need to be put in place for the program? Does the program have activities that occur in the classroom, in another setting at school, at home, or in a combination of these settings?
The READ oversight team examined program materials to determine the primary components of the READ program. They determined that the READ program had three strategies: classroom lessons, homework, and assessments. Each of these strategies required certain activities in order to be successful. For instance, teachers would need professional development on how to integrate the READ classroom lessons into their instruction, as well as how to use the READ assessment data. Students would also need training in how to use the READ system in the classroom and at home.
Strategies might include activities such as professional development, technology access, and the use of curricular materials. Strategies might be ongoing throughout the program or drawn on at various stages during the program’s operation. Listing all strategies and activities used in your program is important to explain later on how and to what extent your program’s goals were met.
