I-IANE~OOI< O1:: EVALUATION M{~TIqODS
Crawford P, Bryce P. Project monitoring and evaluation: a method for enhancing the efficiency and effectiveness of aid project implementation. Int J Proj Manag 2003;21 (5):363-73.
http://jthom.best.vwh.net/usability/ Contains lots o f method overviews with links and references (last visited on 31.05.2005).
155
Brender, McNair, Jytte, and Jytte Brender. Handbook of Evaluation Methods for Health Informatics, Elsevier Science & Technology, 2006. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/waldenu/detail.action?docID=306691. Created from waldenu on 2022-03-06 01:57:27.
C o p yr
ig h t ©
2 0 0 6 . E
ls e vi
e r
S ci
e n ce
& T
e ch
n o lo
g y.
A ll
ri g h ts
r e
se rv
e d .
NANI}8OOK O~ EVALUATION MI~TIdODS
Organizational Readiness
Areas of Application Assessment o f a healthcare organization’s readiness to assimilate a clinical information system.
Description A number of aspects determine the organizational readiness for change, such as organizational adaptability and flexibility, its willingness to absorb external solutions, and its ability to develop viable solutions. A potential cause of failure to innovate is the organizational inability to undergo transformation during the implementation of an information system.
The study of Snyder-Halpern (2001) briefly reviews previous attempts at determining readiness and validates a model of innovation readiness and a set of heuristics to assess organizational readiness. The method is not yet complete and has still to include the metrics o f the heuristics suggested. However, the description may still serve as valuable inspiration to preventive actions through assessment at the early stages o f IT systems purchase or implementation.
Assumptions for Application
Perspectives
Frame of Reference for Interpretation (Not applicable)
Perils and Pitfalls
Advice and Comments
154
Brender, McNair, Jytte, and Jytte Brender. Handbook of Evaluation Methods for Health Informatics, Elsevier Science & Technology, 2006. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/waldenu/detail.action?docID=306691. Created from waldenu on 2022-03-06 01:57:27.
C o p yr
ig h t ©
2 0 0 6 . E
ls e vi
e r
S ci
e n ce
& T
e ch
n o lo
g y.
A ll
ri g h ts
r e
se rv
e d .
HAND~OOI< O~ ~VALUATION N~TIdODS
References
Snyder-Halpem R. Indicators of organizational readiness for clinical information technology/systems innovation: a Delphi study. Int J Med Inform 2001;63(3):179-204. Erratum in: Int J Med Inform 2002;65(3):243.
155
Brender, McNair, Jytte, and Jytte Brender. Handbook of Evaluation Methods for Health Informatics, Elsevier Science & Technology, 2006. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/waldenu/detail.action?docID=306691. Created from waldenu on 2022-03-06 01:57:27.
C o p yr
ig h t ©
2 0 0 6 . E
ls e vi
e r
S ci
e n ce
& T
e ch
n o lo
g y.
A ll
ri g h ts
r e
se rv
e d .
IdANDt~OOI< O~ EVALUATION NE—TIdODS
Pardizipp Areas of Application Preparation of future scenarios
Description Scenarios are common-language descriptions of specific activities and of how users would normally go about executing them. They can, however, also be described diagrammatically. Pardizipp is based on the Delphi method. Development of scenarios, which in Pardizipp are textual, follows the six steps listed below (steps 2-4 are to be repeated jointly) (Mettler and Baumgartner 1998):
1. Definition of a general frame that will serve as the basis for group work around the creation of scenarios
2. Creation of scenarios and a thorough analysis of their consequences and assumptions
3. Quantifying and model building 4. Preparation of policies and specific actions – where a given
scenario should be implemented 5. Development of a consensus scenario, which takes into account
earlier scenarios developed for the same problem area 6. Preparation of recommendations for policies and actual actions
Assumptions for Application A prerequisite for a successful result is that a good mix of stakeholder groups are represented.
It is worth noting that a centrally placed t e a m – not the participants i n v o l v e d – prepares the resulting scenario(s) on the basis of the information gathered. Thus, it requires a certain degree of experience o f this type of undertaking. See also under Delphi.
Perspectives The philosophy behind the method is twofold: It is partly based on the philosophy built into the Delphi method and partly on the fact that modem technological development has a number o f unfortunate effects on surrounding social and organizational conditions. This
ts6
Brender, McNair, Jytte, and Jytte Brender. Handbook of Evaluation Methods for Health Informatics, Elsevier Science & Technology, 2006. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/waldenu/detail.action?docID=306691. Created from waldenu on 2022-03-06 01:57:27.
C o p yr
ig h t ©
2 0 0 6 . E
ls e vi
e r
S ci
e n ce
& T
e ch
n o lo
g y.
A ll
ri g h ts
r e
se rv
e d .
HANDt~OOI< O~ EVALUATION METHODS
makes the authors focus on improving the basis of decision-making and related processes, which is done by establishing a decision- making foundation built on the projections and aspirations o f a broad segment of participants. In other words, they believe that the basis for decision making will improve by expanding the group preparing it. This is obviously a culturally conditioned assumption, see the discussion in the Introduction, and it has its roots in Western cultural perceptions o f how things should be handled and how best to structure an organization.
Frame of Reference for Interpretation (Not applicable)
Perils and Pitfalls 1. The principle of using scenarios gives a fragmented picture of a
normally holistic entirety. Therefore, it takes some talent to organize the preparation o f a combination of scenarios in such a way that together they will give an adequately holistic impression.
2. People normally find it difficult to explicitly formulate their thought about their own work situation and work procedures explicitly (see the discussions in Brender 1997a and 1999 and others). This is where the Delphi method shows its strength, but it cannot completely compensate in all situations.
3. One pitfall is the ethnocentricity (see Part III) – that is, lack of acknowledgement and consideration of cultural backgrounds. This may introduce a bias in the selection of participants. The authors mention principles for participation (for instance, lack o f female involvement) as a caveat, but in general the method is considered to be most appropriate in cultures where there is already a tradition for participatory methods and in organizations where there is a possibility of (informal) equality among participants of different levels of competence.
Advice and Comments Future scenarios are useful tools for the understanding o f development trends and options. Methods for preparing future scenarios often have the advantage of helping the participants to fantasize – irrespective of their technological understanding. This way the participants are often able to let go of actual constraints to the technological potentials and limitations induced by many systems analysis tools.
157
Brender, McNair, Jytte, and Jytte Brender. Handbook of Evaluation Methods for Health Informatics, Elsevier Science & Technology, 2006. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/waldenu/detail.action?docID=306691. Created from waldenu on 2022-03-06 01:57:27.
C o p yr
ig h t ©
2 0 0 6 . E
ls e vi
e r
S ci
e n ce
& T
e ch
n o lo
g y.
A ll
ri g h ts
r e
se rv
e d .
HANDE~OOI( O1::: EVALUATION METHODS
References
Brender J. Methodology for assessment of medical IT-based systems- in an organisational context. Amsterdam: lOS Press, Stud Health Technol Inform 1997;42.
Brender J. Methodology for constructive assessment of IT-based systems in an organisational context. Int J Med Inform 1999;56:67-86.
This is a shortened version o f the first reference and more easily accessible with regard to this subject.
