State tests conducted in April of each academic year
Rubric data collected quarterly, through teacher surveys (all classrooms) and classroom observations (case study classrooms)
Correlational analyses between state achievement test scores and NowPLAN-T rubric scores
T-test of mean test scores pre- NowPLAN-T and each academic year post-NowPLAN-T
Results disaggregated by school, classroom, grade level, gender, race/ethnicity, special education status, and English language proficiency
Improved postsecondary success
To what extent was postsecondary success related to implementation of NowPLAN-T?
Increased correlation between achievement scores and postsecondary success
Note: This component is not currently evaluated. Indicators will be refined and targets will be determined at a later date.
Graduate follow- up surveys and focus groups
Note: This component is not currently evaluated. Data collection will be determined at a later date.
Descriptive statistics
Qualitative analysis of focus group data
143
Logic Model Component
Evaluation Questions
Indicators Targets Data Source Data Collection Data Analysis
Improve d postsecondary success
To what extent was postsecondary success related to student achievement?
Increased correlation between NowPLAN-T implementation and postsecondary success
Note: This component is not currently evaluated. Indicators will be refined and targets will be determined at a later date.
State test scores
Graduate follow- up surveys
Note: This compone nt is not currently evaluated. Data collection will be determi ned at a later date.
Descriptive statistics
Correlational studies and significance testing
144
Step 4: Interpret the Results The NowPLAN-T external evaluator will meet with district staff quarterly to provide interim findings for program improvement and midcourse adjustments. The external evaluator will also provide an annual evaluation report focusing on the district’s progress towards meeting the targets set for NowPLAN-T objectives. Biannual newsletters (traditional and electronic), as well as periodic presentations and press releases, will be used to communicate NowPLAN-T progress and findings to staff, parents, and students.
Step 5: Inform and Refine – USING the Results Evaluation results will be used to inform and improve NowPLAN-T, refine the NowPLAN-T logic model (as necessary), and make recommendations and decisions regarding the future direction of NowPLAN-T.
Additional Notes Embedding Evaluation in the Strategic Plan
Nowgarden School District had been through strategic planning before. Past strategic plans were completed only because they were required and then filed away and rarely consulted. The superintendent, who was hired 3 years ago, saw strategic planning as an opportunity for the district to reflect and grow. The superintendent did not want to ask teachers, administrators, and community members to spend their valuable time participating in a strategic planning process that was not going to be used to its fullest potential. The superintendent knew that a good strategic plan could be used for positive change and growth, and that embedding evaluation within the plan itself would provide information for continuous improvement. District administration agreed that a powerful strategic plan with an embedded evaluation provides the ingredients for success.
The strategic planning team chose to use the NowPLAN rubrics as the cornerstone of its strategic plan. It developed rubrics for each strategy included in the strategic plan. The NowPLAN rubrics were to be used as a guide and benchmarking tool. All teachers received professional development on using the NowPLAN rubrics for self-assessment. Administrators believed that familiarity with the rubrics would provide teachers with an understanding of the district’s expectations (i.e., what the district has determined good practice to “look like”), and that use of the rubrics would encourage self-reflection and ultimately improvement. Embedding the NowPLAN rubrics into everyday practice, from the classroom teacher’s use to the curriculum supervisor’s reviews, was Nowgarden’s way to translate the district’s strategic plan into practice. The Nowgarden superintendent knew that the district’s strategic plan,
145
including its technology plan, could drive change if it was a living, breathing plan that was incorporated as the foundation and into every aspect of the district’s operation.
The superintendent also knew that an important aspect of using rubrics to understand expectations and drive change is consistency. Understanding of the rubrics must be uniform, and application of the rubric must be consistent. For this reason, the external evaluator was asked to compare externally completed observation-based rubric ratings (from case study classrooms) with self-report rubric ratings by classroom teachers. By doing this, the evaluator could uncover discrepancies and inconsistencies in understanding and application. These findings were to be provided to program staff to be used to plan professional development activities that aid in rubric use and to improve the reliability of rubric data.
The evaluator assured the district staff that the reporting of such data would in no way violate teacher confidentiality and privacy. The evaluation team planned to collect and manage data such that individual privacy was maintained. Data were to be stored with “dummy” keys that would allow linkages between data sets, but that would not relate to any internal, district identifier. Only the external evaluator would have access to key coding. Linkages between data sources, such as observational data and survey data, would be performed by the evaluation team and findings would be reviewed prior to release to ensure that individual identities could not be directly or deductively determined.
While summarizations of NowPLAN (including NowPLAN-T) rubric data would be provided to program staff for formative program changes, the evaluator planned to also use rubric data to relate implementation to long-term outcomes. The program theory laid out by district administration assumed that higher rubric scores would be positively related to higher student achievement scores. Similarly, they hypothesized that lower rubric scores (that is, less sophisticated levels of implementation) would be associated with lower student achievement scores.
NowPLAN-T Rubrics Tables 19 through 25 represent the 1:1 Implementation Rubric. The William & Ida Friday Institute for Educational Innovation at North Carolina State University kindly granted permission to reproduce the 1:1 Implementation Rubric in Appendix B. The rubric was developed by research staff at the Friday Institute. The 1:1 Implementation Rubric is based on the International Society for Technology in Education’s National Educational Technology Standards (ISTE’s NETS) framework. It was also developed using the North Carolina IMPACT Guidelines, the Texas Star Chart, and the North Carolina Learning Technology Initiatives (NCLTI) Planning Framework.
The rubric provides an assessment of the daily impact and use of technology programs and services on the teaching and learning process. It can be used to examine technology programs
146
at the district level, as well as the school and classroom levels. The 1:1 Implementation Rubric is intended to aid in reflecting on your technology implementation. For more information, visit https://eval.fi.ncsu.edu/11-implementation-rubric/.
Although the programs, characters, schools, and school district mentioned in Appendix B are fictitious examples provided to illustrate how the principles in this guide can be applied, the 1:1 Implementation Rubric is a real instrument. The 1:1 Implementation Rubric serves as the NowPLAN-T evaluation rubric for the fictitious Nowgarden School District. The NowPLAN-T evaluation incorporates the rubric as a powerful indicator of classroom and school performance. The rubric will be used by teachers for self-assessment and by evaluators during observations of case study classrooms.
The NowPLAN-T evaluation will use the Friday Institute 1:1 Implementation Rubric to examine four dimensions of classroom technology use and teacher experience with technology. The four implementation areas are shown below:
1. Curriculum and Instruction
2. Infrastructure and Technical Support
3. Leadership, Administration, and Instructional Support
4. Professional Development
147
Each of the four implementation areas has six elements of reflection. The NowPLAN-T evaluator created the chart below, using the elements from the Friday Institute 1:1 Implementation Rubric.
Table 19: 1:1 Implementation Rubric: Implementation Areas and Elements of Reflection
Implementation Area Element of Reflection
Curriculum & Instruction (CI) Classroom Use
Access to Digital Content
Content Area Connections
Technology Applications
Student Mastery of Technology Applications
Web-based Lessons
Infrastructure & Technical Support (IA)
Students: Computer
Access/Connectivity
Classroom Technology
Technical Support
LAN/WAN
Student Access to Distance Learning
Leadership, Administration, & Instructional Support (LA)
Leadership and Vision
Planning
Instructional Support
Communication and Collaboration
Sustainability
Policy
Professional Development (PD)
Professional Development Experiences
Models of Professional Development
Educator Capability
Participation in Technology-Driven Professional Development
Levels of Understanding
Student Training
148
