Deck 17: Evaluation Research

Question 1

Match each of the following examples of educational evaluation with the purpose for conducting an evaluation that it illustrates.

-The developer of an innovative approach to teaching English to migrant children is asked for documentation of the success rates with this approach.

A) A guide to policy decisions in education
B) A basis for advocacy for political action
C) A component of the management of educational programs
D) Use of research to design new programs meeting specified criteria of effectiveness

Answer 1

Use of research to design new programs meeting specified criteria of effectiveness

Question 2

Match each of the following examples of educational evaluation with the purpose for conducting an evaluation that it illustrates.

-A high school principal asks an evaluator to collect data on the costs and benefits of the school's efforts to reduce student dropout.

A) A guide to policy decisions in education
B) A basis for advocacy for political action
C) A component of the management of educational programs
D) Use of research to design new programs meeting specified criteria of effectiveness

Answer 2

A component of the management of educational programs

Question 3

Match each of the following examples of educational evaluation with the purpose for conducting an evaluation that it illustrates.

-Opponents of site-based management cite negative evaluations of this approach in a public hearing.

A) A guide to policy decisions in education
B) A basis for advocacy for political action
C) A component of the management of educational programs
D) Use of research to design new programs meeting specified criteria of effectiveness

Answer 3

A basis for advocacy for political action

Question 4

Match each of the following examples of educational evaluation with the purpose for conducting an evaluation that it illustrates.

-A researcher collects data to prepare a position paper on the costs, benefits, and problems of various means to promote adult literacy.

A) A guide to policy decisions in education
B) A basis for advocacy for political action
C) A component of the management of educational programs
D) Use of research to design new programs meeting specified criteria of effectiveness

Question 5

The main reason that educational evaluation has attracted increased interest is

A) the unique nature of the research designs used in evaluation research.
B) the important role evaluation plays in policy analysis and program management.
C) increased public demands for educational accountability.
D) development of standards for program evaluation.

Question 6

Evaluation research differs most from other types of educational research in its emphasis on

A) producing generalizable findings.
B) demonstrating statistically significant differences or relationships between variables.
C) yielding data that will facilitate decisions about an educational phenomenon.
D) obtaining the opinions of individuals about educational phenomena.

Question 7

The most important factor in working with stakeholders when evaluating an educational program is to

A) identify all groups and individuals with a potential interest in the program.
B) develop a plan that allows all interested stakeholders to collaborate in the evaluation.
C) determine and provide for different types of stakeholders the amount of involvement that they desire.
D) assign different stakeholder groups to participate in evaluation of those program aspects with which they are most familiar.

Question 8

An outside evaluator is hired to evaluate a school program that has special funding. A teacher tells him that he actually was brought in to collect data to justify the principal's plan to reassign several staff members.The evaluator's most appropriate action in this situation is to

A) interview a variety of key individuals in the field setting to explore further the basis for the evaluation request.
B) ask the teacher to provide evidence to support her claim.
C) terminate the evaluation effort in the interests of ethical principles.
D) ask the principal to reconfirm the previously stated purpose of the evaluation.

Question 9

The outcome of program delineation is

A) analysis of the program to determine the components for which evaluation data are available.
B) the selection of the specific model of evaluation to be used to evaluate the program.
C) the determination of the effectiveness of the program that is evaluated.
D) an identification of the most important characteristics of the program that is to be evaluated.

Question 10

An example of a program resource is the

A) set of criteria used to hire staff for the program.
B) objectives that program staff are committed to achieve.
C) observed benefits of the program.
D) staff needed to implement the program.

Question 11

Match each question with the component of an educational program that needs to be evaluated to answer it.

-How will students have changed following completion of the program?

A) Program goals
B) Program resources
C) Program procedures
D) Program management
E) Program outcomes

Question 12

Match each question with the component of an educational program that needs to be evaluated to answer it.

-What are the obvious and hidden costs of the program?

A) Program goals
B) Program resources
C) Program procedures
D) Program management
E) Program outcomes

Question 13

Match each question with the component of an educational program that needs to be evaluated to answer it.

-What school improvements are anticipated from implementation of this program?

A) Program goals
B) Program resources
C) Program procedures
D) Program management
E) Program outcomes

Question 14

Match each question with the component of an educational program that needs to be evaluated to answer it.

-To what extent did teachers actually use the techniques recommended in the program?

A) Program goals
B) Program resources
C) Program procedures
D) Program management
E) Program outcomes

Question 15

Match each question with the component of an educational program that needs to be evaluated to answer it.

-Did the principal's supervision of the teachers help them improve their use of program techniques?

A) Program goals
B) Program resources
C) Program procedures
D) Program management
E) Program outcomes

Question 16

Gathering a comprehensive list of evaluation questions from stakeholders is the______ phase in selecting questions for an evaluation study.

A) program delineation
B) divergent
C) goal definition
D) convergent

Question 17

The convergent phase of an evaluation involves

A) the evaluators' reduction of all possible evaluation questions being considered to a manageable number.
B) stakeholders' determination of the final foci for the evaluation.
C) stakeholder's identification of additional questions to be considered for evaluation.
D) The evaluators' development of an evaluation time line.

Question 18

An important question to consider in creating a design for a program evaluation is,

A) Which stakeholders should conduct the evaluation?
B) How can the evaluation be conducted so as not to incur any extra costs?
C) Can an outside evaluator be found who has a major stake in the program?
D) How much internal and external validity can be built into the research design?

Question 19

Probably the most common type of quantitative research design used in evaluation studies is the______ research design.

A) correlational
B) single-case
C) experimental
D) causal-comparative

Question 20

The Standards for Evaluations of Educational Programs, Projects, and Materials were developed to

A) disseminate information about the exemplary quality of the standards currently used in evaluation research.
B) reflect the varied political orientations of individuals working in the field of evaluation.
C) assemble into one source several sets of standards that various research and educational organizations had thoroughly tested in the field.
D) improve the professionalism of educational evaluation.

Question 21

The main focus of the Standards for Evaluations of Educational Programs, Projects, and Materials is

A) optimizing the instrumental use of evaluation research.
B) selecting the best strategy for assessment of educational personnel.
C) assessing the feasibility of including an evaluation component in a program design.
D) comparing individuals' test performance with the appropriate norms.

Question 22

The data collection and analysis phases in an evaluation study

A) are typically much more extensive than in other types of research.
B) involve primarily quantitative measures of a program's outcome.
C) are similar to that in other types of research.
D) are usually carried out by an external evaluator.

Question 23

The report of an evaluation study typically differs from the report of a research study in that the former

A) is simpler, because stakeholders only want the "bottom line."
B) is often prepared in different forms for different stakeholder groups.
C) contains a more extensive review of the literature.
D) places more emphasis on the generalizability of the findings.

Question 24

A meta-evaluation is

A) similar to a pilot test of a research study.
B) a summary of any weaknesses in the research design or execution.
C) a description of the extent to which the evaluation findings can be generalized.
D) an assessment by stakeholders of how well the evaluation was carried out.

Question 25

Evaluation research as a special application of educational research can be traced back most clearly to

A) the growing interest in qualitative research.
B) research and development (R&D) in industry and medicine.
C) the need for public funds to support compulsory education.
D) the individual intelligence testing movement.

Question 26

Goal-free evaluation

A) helps alert developers to adverse effects of the program.
B) typically is carried out in the early stages of program development.
C) involves collecting primarily data that are unrelated to the program's stated goals.
D) is a collaborative effort between the evaluator and key stakeholders.

Question 27

Goal-free evaluation differs from other approaches to evaluation mainly in that it

A) requires the evaluator to be uninformed of the intended goals of the program.
B) requires unique statistical techniques.
C) is used to evaluate programs in the formative stage.
D) is concerned primarily with negative program effects.

Question 28

The main rationale for the development of objectives-based evaluation was that

A) information about individual differences in student aptitude is needed for selection decisions.
B) students might perform poorly in school not because of a lack of innate ability but because of weaknesses in the curriculum.
C) the merit of an educational program must be assessed in relation to the values and beliefs of different stakeholders.
D) it is difficult to specify the most desired instructional outcomes in behavioral terms.

Question 29

The goal of a college education for all citizens who desire one is an example of a need stated as

A) an expectation.
B) a norm.
C) a minimum.
D) an ideal.

Question 30

Match each example of the findings of evaluation research listed below with the type of evaluation in the CIPP model that it illustrates.

-Recruitment programs should be implemented for doctoral students who have the desired characteristics to become special education faculty members.

A) Context evaluation
B) Input evaluation
C) Process evaluation
D) Product evaluation

Question 31

Match each example of the findings of evaluation research listed below with the type of evaluation in the CIPP model that it illustrates.

-The recruitment program has succeeded in attracting more Hispanic graduate students in special education, but the proportion of African-American students recruited has not changed.

A) Context evaluation
B) Input evaluation
C) Process evaluation
D) Product evaluation

Question 32

Match each example of the findings of evaluation research listed below with the type of evaluation in the CIPP model that it illustrates.

-Not enough special education students are receiving doctorates to fill available faculty positions in special education.

A) Context evaluation
B) Input evaluation
C) Process evaluation
D) Product evaluation

Question 33

Match each example of the findings of evaluation research listed below with the type of evaluation in the CIPP model that it illustrates.

-Attendance at recruitment program meetings designed to attract more minority students into special education revealed that a lower proportion of African-Americans and Hispanics attend the programs than their representation in the general graduate student body at this university.

A) Context evaluation
B) Input evaluation
C) Process evaluation
D) Product evaluation

Question 34

Match each of the procedures below with the type of educational evaluation that uses this procedure.

-The degree of congruence between program standards and actual program performance is determined.

A) Goal-free evaluation
B) Context-Input-Process-Product evaluation
C) Needs assessment
D) Cost analysis

Question 35

Match each of the procedures below with the type of educational evaluation that uses this procedure.

-The relationship between program benefits and the resources needed to achieve those benefits is calculated.

A) Goal-free evaluation
B) Context-Input-Process-Product evaluation
C) Needs assessment
D) Cost analysis

Question 36

Match each of the procedures below with the type of educational evaluation that uses this procedure.

-Data are collected about present needs, required resources and strategies, program operation, and goal achievement.

A) Goal-free evaluation
B) Context-Input-Process-Product evaluation
C) Needs assessment
D) Cost analysis

Question 37

Match each of the procedures below with the type of educational evaluation that uses this procedure.

-Evaluators conduct research to discover the actual effects of the program in operation.

A) Goal-free evaluation
B) Context-Input-Process-Product evaluation
C) Needs assessment
D) Cost analysis

Question 38

Which of the following is a characteristic of responsive evaluation?

A) It makes extensive use of subjective inquiry.
B) It focuses on objective response patterns.
C) It refers to psychological theories of development.
D) A research design is first developed and the evaluation responds to the demands of that design.

Question 39

A desirable requirement for a graduate student wishing to use responsive evaluation as a model for an evaluation study for a dissertation is that the student should

A) be experienced in carrying out a variety of research designs.
B) work with an experienced team of evaluators on a joint project.
C) study a stratified sample of stakeholders.
D) be a stakeholder in the program to be evaluated.

Question 40

Which of these evaluation models involves independent development of pro and con arguments concerning program-related issues?

A) The Context-Input-Process-Product model
B) Discrepancy evaluation
C) Adversary evaluation
D) Goal-free evaluation

Question 41

An important difference between adversary evaluation and judicial evaluation is that the former

A) involves a public hearing in which witnesses testify before a panel that makes recommendations..
B) is based on the use of legal procedures to review program characteristics.
C) involves a debate between two evaluation teams, with victory or persuasion as the desired outcome.
D) has been widely used in educational research.

Question 42

Unlike other qualitative models of evaluation, educational connoisseurship and criticism is unique in that

A) the questions that motivate the evaluation are set by the evaluator alone.
B) it combines both qualitative and quantitative research methods.
C) the evaluative orientation toward the phenomena being studied is based on critical theory.
D) it is based largely on the research tradition of cognitive anthropology.

Question 43

Empowerment evaluation differs most from other qualitative models of evaluation primarily in

A) the prominent role of stakeholders in the evaluation design.
B) the use of the latest technological tools to facilitate data collection and analysis.
C) its focus on establishing the mission or vision of the community as a basis for program improvement.
D) all of the above.

Question 44

Responsive evaluation models and objectives-based evaluation models differ on several important characteristics. Place an R in front of the characteristics of responsive evaluation and an 0 in front of the characteristics of objectives-based evaluation in the following list.

-Uses a formal rather than an informal approach.

A) R
B) 0

Question 45

Responsive evaluation models and objectives-based evaluation models differ on several important characteristics. Place an R in front of the characteristics of responsive evaluation and an 0 in front of the characteristics of objectives-based evaluation in the following list.

-Focuses on identifying concerns and issues.

A) R
B) 0

Question 46

Responsive evaluation models and objectives-based evaluation models differ on several important characteristics. Place an R in front of the characteristics of responsive evaluation and an 0 in front of the characteristics of objectives-based evaluation in the following list.

-Is modeled after the procedures of experimental psychology.

A) R
B) 0

Question 47

Responsive evaluation models and objectives-based evaluation models differ on several important characteristics. Place an R in front of the characteristics of responsive evaluation and an 0 in front of the characteristics of objectives-based evaluation in the following list.

-Employs careful preplanning of the evaluation design.

A) R
B) 0

Question 48

Responsive evaluation models and objectives-based evaluation models differ on several important characteristics. Place an R in front of the characteristics of responsive evaluation and an 0 in front of the characteristics of objectives-based evaluation in the following list.

-Involves purposeful sampling of stakeholders.

A) R
B) 0

Question 49

Formative evaluation

A) involves collecting data about an educational program while it is still under development in order to improve it.
B) enables the researcher to generalize the findings to other programs and samples.
C) involves testing research hypotheses about a program prior to submitting it to summative evaluation.
D) is conducted by using primarily quantitative research methods.

Question 50

The three-level process of formative evaluation described in the text

A) corresponds closely to the evaluation procedures used by major textbook publishers.
B) moves progressively from a focus on qualitative methods to a focus on quantitative methods of evaluation.
C) moves progressively from a focus on quantitative methods to a focus on qualitative methods of evaluation.
D) involves a review of the program by stakeholders at each stage.

Question 51

In contrast to formative evaluation, summative evaluation

A) involves a critical review of each program component by key stakeholders.
B) is carried out by an evaluator working alone.
C) is considered an integral part of the instructional design process.
D) is likely to involve quantitative research methods.

Question 52

An important purpose of summative evaluation is to

A) help developers determine what the goals of a program should be.
B) determine how successful the program is under normal operating conditions.
C) discover the goals of a program by observing the program while it is being developed.
D) identify unintended outcomes for which the program has application.

Question 53

A common criticism of behavioral objectives is that they tend to

A) omit the situational context in which the behavior is to be performed.
B) not give stakeholders an adequate sense of what those who succeed as a result of the program will be able to do.
C) not provide much help in designing test items to evaluate program effectiveness.
D) reduce education to a matter of teaching only that which is easily measured.

Question 54

Behavioral objectives are particularly helpful in

A) determining the level of resources needed to develop an instructional program.
B) determining the causes of students' failure to learn from an instructional program.
C) designing test items to assess students' capabilities before and after experiencing an instructional program.
D) one-on one testing of hypotheses derived from instructional design theory.

Question 55

Educational research and development holds great promise for improving education primarily because it

A) requires collaboration between a variety of stakeholder groups in the evaluation.
B) requires the use of behavioral objectives to evaluate student learning.
C) provides a great deal of evaluation information for a relatively low cost.
D) involves a close connection between systematic program evaluation and program development.

Question 56

A good way for a graduate student to design an educational research and development project to make it more manageable to conduct is to

A) serve as both the developer and the evaluator for the project.
B) compare the effectiveness of two alternate versions of the developed program.
C) end the dissertation study after the preliminary field test stage.
D) use only qualitative methods of evaluation.

Question 57

Describe two purposes for which educational evaluation studies are carried out.

Question 58

The chapter lists eight steps in program evaluation, beginning with clarifying the reasons for an evaluation. List three of the other steps of a program evaluation.

Question 59

Briefly state one of the differences between educational evaluation and educational research.

Question 60

If evaluators are asked to conduct an evaluation, what procedure should they use to clarify the reasons for doing the evaluation?

Question 61

(a) Define the term stakeholder as used in evaluation research, and (b) give two reasons why stakeholders need to be identified as part of an evaluation study.

Question 62

List two of the procedures that should be followed in determining which aspects of the overall program should be evaluated.

Question 63

(a) Identify the five key components of a program to be evaluated that the chapter describes, and (b) explain what one of them involves.

Question 64

What is the difference between the divergent phase and the convergent phase in selecting questions for an evaluation study?

Question 65

Briefly describe two of the questions that should be considered in developing an evaluation design.

Question 66

What is one negative effect that can occur from the evaluation of an educational program?

Question 67

The Standards for Evaluations of Educational Programs, Projects, and Materials developed four major criteria for good evaluation studies: utility, feasibility, propriety, and accuracy. Describe one specific standard for each of these broad criteria.

Question 68

Describe one type of data-collection method that might be useful in the evaluation of an instructional program.

Question 69

Describe two desirable features of a dissertation involving educational evaluation.

Question 70

(a) Define meta-evaluation; (b) explain what a meta-evaluation typically includes; and (c) indicate whether meta-evaluation is necessary only in evaluation research, only in other types of educational research, or in both evaluation research and other types of research.

Question 71

What is the main difference between the evaluation of individuals and objectives-based evaluation?

Question 72

Name two possible problems associated with objectives-based evaluation.

Question 73

What is meant by goal-free evaluation?

Question 74

State one problem that can arise in doing a needs assessment.

Question 75

Briefly describe the four types of educational evaluation that are incorporated in the CIPP evaluation model.

Question 76

What is the main purpose of conducting a formative evaluation of an educational program?

Question 77

What is the main way in which qualitative approaches to evaluation differ from quantitative approaches?

Question 78

Describe the four stages of adversary evaluation.

Question 79

Describe one shortcoming of adversary evaluation.

Question 80

In the qualitative evaluation model known as educational connoisseurship and criticism, describe (a) what connoisseurship involves and (b) what criticism involves.