ANDt~OOI< O~ EVALUATION METHODS
based solution, it makes a huge difference to how you proceed. Please see the list of references with regard to additional descriptions of procedures.
Assumptions for Application It is obvious that a measurement of fulfillment of the objective assumes that a strategic objective has been defined and can be made operational in terms of relevant and realistic m e a s u r e s – that is, that formulas (metrics) can be established for their measurement.
Perspectives Few people realize the wide range of consequences an IT-based solution can have even beyond the organization normally considered as the user of the system: On implementation of a (new) IT system, it is nearly always necessary to make compromises (a bit of give and take). This is not necessarily bad, as over time many organizations develop very specific ways of doing things in a never-ending number of variations that often cannot be handled by the IT system. Some forms o f rationalization will therefore typically result from the implementation of an IT system. This rationalization will impact on something or somebody, and therefore there is a risk that the changes within the user organization will reach far and wide into the surrounding organizations.
Once introduced into daily operation, the organization keeps on c h a n g i n g – initially as it gets accustomed to and adjusts to the system. One danger (and a pitfall) is to start measuring the effect too soon after the daily operation has started before a degree of stability has been achieved. On the other hand, after a certain period in operation staff will start using their creativity, and new ways of doing things will develop spontaneously- both appropriate ones and inappropriate ones, intentional ones and unintentional ones (as shown in van Gennip and Bakker 1995 or in Part III, Section 184.108.40.206). This phenomenon only reflects the development pattern from novice to expert: You start by following the prescribed way of doing things, but while gaining experience at a proficient level, you get to know the shortcuts (Dreyfus 1997). You could, for instance, compare the note-taking in patient records by a newly graduated physician to that of a consultant. This phenomenon may make comparisons of controlled studies and before and after studies rather difficult.
Frame of Reference for Interpretation The strategic objective may influence all aspects of an organization,
Brender, McNair, Jytte, and Jytte Brender. Handbook of Evaluation Methods for Health Informatics, Elsevier Science & Technology, 2006. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/waldenu/detail.action?docID=306691. Created from waldenu on 2022-03-06 01:57:27.
C o p yr
ig h t ©
2 0 0 6 . E
ls e vi
e n ce
n o lo
ri g h ts
e d .
~ANI38OOK Og EVALUATION M~TNODS
right from (partly) financially, matters o f responsibility and competence, quality objectives for patient care or the overall service, overall to the number o f records missing or patient and staff satisfaction. Whether there is a need for before-values (baseline data) depends on the individual study.
IT systems that do not have a written defined strategic objective could possibly have an objective defined at a lower, tactical level (for instance, in terms o f the interim objectives that are expected to lead to the overall strategic objective), and this may perfectly well replace the strategic objective as a frame o f reference or they can form the frame o f reference together.
The frame o f reference must therefore be aligned with the real objective and to the conditions o f the actual study. In order to describe the conditions adequately, the objective will usually need to be covered by a whole range o f indicators (measures).
Perils and Pitfalls A consequence o f the extremely dynamic situations occurring in connection with the introduction o f an IT-based system could be the loss o f control o f what some o f the effect indicators actually signify. Before-and-after assessment, as, for instance, in those recommended in (Nyttevcerdi 2002), have inbuilt pitfalls in the shape o f the validity o f the frame o f reference. One must be sure that the concepts used are the same before and after in case they are directly measured and compared. It is often necessary to use different questionnaires for the before-and-after situations, as the work proces