In another case, the quality of a hospital could be assessed by looking at its spe- cial programs, its operating facilities, its emergency room operations, its in-patient operations, its pharmacy, and so on, by experts in medicine, health services, and hospital administration. They could examine facilities and equipment/supplies of the hospital, its operational procedures on paper and in action, data on the fre- quency and outcomes of different procedures, the qualifications of its personnel, patient records, and other aspects of the hospital to determine whether it is meeting appropriate professional standards.
Although professional judgments are involved to some degree in all evalua- tion approaches, this one is decidedly different from others because of its direct, open reliance on professional expertise as the primary evaluation strategy. Such expertise may be provided by an evaluator or by subject-matter experts, depend- ing on who might offer most in the substance or procedures being evaluated. Usually one person will not own all of the requisite knowledge needed to adequately evaluate the program, institution, or agency. A team of experts who complement each other are much more likely to produce a sound evaluation.
Several specific evaluation processes are variants of this approach, including doctoral examinations administered by a committee, proposal review panels, site visits and conclusions drawn by professional accreditation associations, reviews of institutions or individuals by state licensing agencies, reviews of staff performance for decisions concerning promotion or tenure, peer reviews of articles submitted to professional journals, site visits of educational programs conducted at the behest of the program’s sponsor, reviews and recommendations by prestigious blue-ribbon
128 Part II • Alternative Approaches to Program Evaluation
TABLE 5.1 Some Features of Four Types of Expertise-Oriented Evaluation Approaches
Type of Expertise-Oriented Evaluation Approach
Existing Structure
Published Standards
Specified Schedule
Opinions of Multiple Experts
Status Affected by Results
Formal review system Yes Yes Yes Yes Usually
Informal review system Yes Rarely Sometimes Yes Usually
Ad hoc panel review No No No Yes Sometimes
Ad hoc individual review No No No No Sometimes
panels, and even the critique offered by the ubiquitous expert who serves in a watchdog role.
To impose some order on the variety of expertise-oriented evaluation activ- ities, we have organized and will discuss these manifestations in four categories: (1) formal professional review systems, (2) informal professional review systems, (3) ad hoc panel reviews, and (4) ad hoc individual reviews. Differences in these categories are shown in Table 5.1, along the following dimensions:
1. Is there an existing structure for conducting the review? 2. Are published or explicit standards used as part of the review? 3. Are reviews scheduled at specified intervals? 4. Does the review include opinions of multiple experts? 5. Do results of the review have an impact on the status of whatever is being
evaluated?
Developers of the Expertise-Oriented Evaluation Approach and Their Contributions
It is difficult to pinpoint the origins of this approach, since it has been with us for a very long time. It was formally used in education in the 1800s, when schools be- gan to standardize college entrance requirements. Informally, it has been in use since the first time an individual to whom expertise was publicly accorded ren- dered a judgment about the quality of some endeavor—and history is mute on when that occurred. Several movements and individuals have given impetus to the various types of expertise-oriented evaluations.
Elliot Eisner, an early evaluator discussed later in this chapter, stressed the role of connoisseurship and criticism in evaluation, roles that required exper- tise in the subject matter to be evaluated. James Madison and Alexander Hamilton took on the role of “expert evaluators” in discussing and elaborating on the meaning and merits of the newly proposed Constitution in The Federalist Papers. (They were experts because they were both present and active at the Constitutional Convention that drafted the document. As such, they were also
Chapter 5 • First Approaches: Expertise and Consumer-Oriented Approaches 129
internal evaluators!) Their writings were influential at the time and are still used by jurists in the U.S. courts to interpret the meanings of the Constitution, illustrating the important actions that can come from reasoned judgments by experts about a product. Accreditation of institutions of higher education is the primary present-day application of expertise-oriented evaluations. The New England Association of Schools and Colleges, which granted the first accredita- tion and continues accreditations for colleges and universities in New England today, began in 1885 when a group of headmasters of preparatory secondary schools began meeting with presidents of colleges in New England to discuss what graduates should know to be prepared for college. Thus, more than 100 years ago, school and college leaders were talking about ways to align their curricula!
Formal Professional Review Systems: Accreditation
Historical Foundations. To many, the most familiar formal professional review system is that of accreditation, the process whereby an organization grants approval of institutions such as schools, universities, and hospitals. Beginning in the late 1800s, regional accreditation agencies in the United States gradually supplanted the borrowed western European system of school inspections. These agencies became a potent force in accrediting institutions of higher education during the 1930s. Education was not alone in institutionalizing accreditation processes to determine and regulate the quality of its institutions. Parallel efforts were under way in other professions, including medicine and law, as concern over quality led to wide-scale acceptance of professionals judging the efforts of those educating fellow professionals. Perhaps the most memorable example is Flexner’s (1910) examination of medical schools in the United States and Canada in the early 1900s, which led to the closing of numerous schools he cited as inferior. As Floden (1983) has noted, Flexner’s study was not accreditation in the strict sense, because medical schools did not participate voluntarily, but it certainly qualified as accreditation in the broader sense: a classic example of pri- vate judgment evaluating educational institutions.
Flexner’s approach differed from most contemporary accreditation efforts in two other significant ways. First, Flexner was not a member of the profession whose efforts he presumed to judge. An educator with no pretense of medical ex- pertise, Flexner nonetheless ventured to judge the quality of medical training in two nations. He argued that common sense was perhaps the most relevant form of expertise:
Time and time again it has been shown that an unfettered lay mind, is . . . best suited to undertake a general survey. . . . The expert has his place, to be sure; but if I were asked to suggest the most promising way to study legal education, I should seek a layman, not a professor of law; or for the sound way to investigate teacher training, the last person I should think of employing would be a professor of education. (Flexner, 1960, p. 71)
130 Part II • Alternative Approaches to Program Evaluation
It should be noted that Flexner’s point was only partially supported by his own study. Although he was a layman in terms of medicine, he was an educator, and his judgments were directed at medical education rather than the practice of medicine, so even here appropriate expertise seemed to be applied.
Second, Flexner made no attempt to claim empirical support for the criteria or process he employed, because he insisted that the standards he used were the “obvious” indicators of school quality and needed no such support. His methods of collecting information and reaching judgments were simple and straightforward: “A stroll through the laboratories disclosed the presence or absence of apparatus, museum specimens, library, and students; and a whiff told the inside story regarding the manner in which anatomy was cultivated” (p. 79).
Third, Flexner dispensed with the professional niceties and courteous criti- cisms that often occur in even the negative findings of today’s accreditation processes. Excerpts of his report of one school included scathing indictments such as this: “Its so-called equipment is dirty and disorderly beyond description. Its outfit in anatomy consists of a small box of bones and the dried-up, filthy frag- ments of a single cadaver. A cold and rusty incubator, a single microscope, . . . and no access to the County Hospital. The school is a disgrace to the state whose laws permit its existence” (Flexner, 1910, p. 190).
