Acknowledgements This guide was created with the valuable input and advice from many individuals. Some individuals provided input into shaping the initial conceptual framework of the guide, some in editing portions of the guide, and some in reviewing draft versions of the guide.
Kathleen Barnhart, Principal Education Consultant, Illinois State Board of Education
Barbara DeCarlo, Retired Principal and Teacher
Beverly Funkhouser, Adjunct Professor, University of Delaware
Rick Gaisford, Educational Technology Specialist, Utah State Office of Education
Robert Hampel, Interim Director, School of Education, University of Delaware
Vic Jaras, Education Technology Director, Iowa Department of Education
Karen Kahan, Director of Educational Technology, Texas Education Agency
Tonya Leija, Reading Recovery Teacher Leaders, Spokane Public Schools
Melinda Maddox, Director of Technology Initiatives, Alabama Department of Education
Daniel Maguire, District Instructional Technology Coach, Kennett Consolidated School District
Jeff Mao, Learning Technology Policy Director, Maine Department of Education
Jennifer Maxfield, Research Associate, Friday Institute for Educational Innovation, North Carolina State University
Brandy Parker, Graduate Research Assistant, Friday Institute for Educational Innovation, North Carolina State University
Shannon Parks, State Education Administrator, Technology Initiatives, Alabama Department of Education
Barry Tomasetti, Superintendent, Kennett Consolidated School District
Bruce Umpstead, Educational Technology Director, Michigan Department of Education
Carla Wade, Technology Education Specialist, Oregon Department of Education
Brent Williams, Director, Educational Technology Center, Kennesaw State University
iv
Thanks also to Jeni Corn, Director of Evaluation Programs, with the William & Ida Friday Institute for Educational Innovation at North Carolina State University for obtaining approval to use the 1:1 Implementation Rubric in the Evaluation Matters guide.
I would like to extend a special thank you to Jenelle Leonard, Director of School Support and Rural Programs (SSRP), Office of Elementary and Secondary Education (OESE) at the U.S. Department of Education, for being the driving force in the creation of this guide.
In addition, I would like to thank Andy Leija, Kelly Bundy, Kim Blessing, Janelle McCabe, and Anna Morgan with Kauffman & Associates, Inc. for their continued support during the creation of the guide.
And finally, I would like to especially thank Nancy Loy (SSRP/OESE) at the U.S. Department of Education for her constant assistance and support throughout the development and writing of the guide, from brainstorming ideas to reading multiple drafts to facilitating review of the evaluation guide.
Portions of this guide were adapted from An Educator’s Guide to Evaluating the Use of Technology in Schools and Classrooms prepared by Sherri Quiñones and Rita Kirshstein at the American Institutes for Research for the U.S. Department of Education in 1998 (Nancy Loy, Project Officer).
v
Before You Get Started Some who use this guide, especially those who are unfamiliar with evaluation or educational program design, may decide to read it cover to cover. However, most who read the guide will likely use it as a compendium and a companion with which they will travel to those portions that are relevant to their current needs. To facilitate this use, there are several features that will aid you in your navigation through the guide.
Click on the I note icon to go to excerpts from Appendix A: Embedded Evaluation Illustration – READ* that appear throughout the text to illustrate each step of the
evaluation process. If you find the excerpts interspersed within text to be distracting, you may want to skip them in the main text and instead read the example in its entirety in Appendix A. There you will find a detailed example of a theory-driven, embedded program evaluation from its inception through the use of its first-year results. Appendix B: Embedded Evaluation Illustration – NowPLAN* provides another example. Both examples set out in this guide are provided solely for the purpose of illustrating how the principles in this guide can be applied in actual situations. The programs, characters, schools, and school districts mentioned in the examples are fictitious.
Click on the R note icon to see additional resources on a topic included in Appendix C: Evaluation Resources.
vi
Introduction What Is the Purpose of the Guide?
Who Is this Guide For? This guide is written for educators. The primary intended audience is state- and district-level educators (e.g., curriculum supervisors, district office personnel, and state-level administrators). Teachers, school administrators, and board members also may find the guide useful. It is intended to help you build evaluation into the programs and projects you use in your classrooms, schools, districts, and state. This guide will also provide a foundation in understanding how to be an informed, active partner with an evaluator to make sure that evaluation provides the information you need to improve the success of your program, as well as to make decisions about whether to continue, expand, or discontinue a program.
No previous evaluation knowledge is needed to understand the material presented. However, this guide may also be useful for experienced evaluators who want to learn more about how to incorporate theory-based evaluation methods into their programs and projects.
In addition to using the guide to embed evaluation within your program, the guide will be useful for
•
•
•
•
•
State education agencies during preparation of program and evaluation guidelines within Requests for Proposals (RFPs), in order to facilitate uniform assessments of proposals and for districts to know how their proposals will be assessed.
School districts in responding to RFPs or in writing grant proposals, in order to set clear expectations for what a program is intended to accomplish and how the evaluation will be embedded within the program to measure changes as a result of the program.
Teams of educators to show value added for a program, in order to build program support and provide budget justification.
Program staff to tell the story of a program using data.
Organizations for evaluation training and professional development
1
How Is this Guide Different From Other Evaluation Guides? There are many evaluation guidebooks, manuals, and tool kits readily available. So, what makes the material presented in this guide different from other evaluation guides? This guide is written with you, the educator, in mind. It outlines an evaluation approach that can be built
into your everyday practice. It recognizes the preciousness of time, the need for information, and the tension between the two. The theory- driven, embedded approach to evaluation is not an additional step to be superimposed upon what you do and the strategies you use but rather a way to weave evaluation into the design, development, and implementation of your programs and projects.
The term program is used broadly in this guide to represent activities, small interventions, classroom-based projects, schoolwide programs, and district or statewide initiatives.
2
This guide will help you to embed evaluation within your program in order to foster continuous improvement by making information and data the basis upon which your program operates. The step-by-step approach outlined in this guide is not simply a lesson in “how to evaluate” but rather a comprehensive approach to support you in planning and understanding your program, with a rigorous evaluation included as an integral part of your program’s design.
In Appendices A and B, you will find two examples of educators building evaluation into their everyday practices. Through a narrative about programs, characters, schools, and school districts that are fictitious, each example is designed to illustrate how the principles in this guide can be applied in actual situations. While embedded evaluation can be used for any type of program you may be implementing, these illustrations specifically focus on programs that involve infusing technology into the curriculum in order to meet teaching and learning goals.
Why Evaluate and What Do I Need to Consider?
Why Evaluate? Evaluation is important so that we can be confident the programs we are using in our schools and classrooms are successful. A common criticism regarding evaluation is that it takes time and resources that could be dedicated to educating students. However, evaluation, done properly, can actually result in better quality practices being delivered more effectively to enhance student learning.
You would not hire new teachers without regularly monitoring and mentoring to help them improve their skills and foster student success. Would you adopt and maintain a new curriculum full scale without being sure that student learning improved when you tested the
new curriculum? What if student learning declined after implementing a new curriculum? How would you know whether the curriculum did not work well because it was a faulty curriculum,
or because teachers were not trained in how to use the curriculum, or because the curriculum was not implemented properly? Building evaluation into your educational programs and strategies enables you to make midcourse corrections and informed decisions regarding whether a program should be continued,
expanded, scaled down, or discontinued.
Evaluation enables you to identify and use better quality practices more effectively to improve learning outcomes.
3
