Deck 2: Planning an Evaluation

Question 1

The first step in planning an evaluation is to

A) suggest outcome measurement instruments to the staff.
B) identify the stakeholders of the evaluation.
C) try to figure out how the evaluation could be done as an experiment.
D) caution the program director that the evaluation will be tough.

Answer 1

B

Question 2

Evaluators must be very careful to be sure that they understand the actual purpose of a request for an evaluation because

A) many stakeholders talk about evaluation in terms that sound like summative evaluation when they really want to improve the program.
B) most program staff members are too willing to disrupt the service program to accommodate the evaluation.
C) the demand for evaluation is so widespread that program staff are eager to participate.
D) stakeholders are eager to apply all recommendations from an evaluator.

Answer 2

A

Question 3

Program evaluation, like other types of applied social science research, differs from basic research in that

A) validity is irrelevant in applied research although reliability is critical to both.
B) reliable measurement is irrelevant in applied research.
C) there are definite time limits for completing applied research projects.
D) program participants cannot refuse to provide data.

Answer 3

C

Question 4

A skill that is more important in program evaluation compared to basic social science research is

A) estimating accurately how much time each phase of the project will require.
B) being able to plan valid research designs.
C) selecting measures of behavior that are reliable.
D) molding the project to answer questions relevant to social theories.

Question 5

"Evaluability assessment" refers to the

A) ability of the program managers to fund an evaluator.
B) knowing how to carry out statistical analyses.
C) reliability of the measures of program outcome that are chosen.
D) likelihood that a valid evaluation can be completed.

Question 6

A major failing of many evaluations is that the evaluator never learned

A) what the outcome measures were.
B) why the program activities were expected to lead to the desired outcomes.
C) who was to receive the report and use the findings.
D) how the analysis was done or what the statistical findings were.

Question 7

Understanding the conceptual foundation of a program provides an evaluator with

A) hints about how the program services are expected to lead to the outcomes that the staff and director hope to achieve.
B) some indication of how much impact the evaluation will actually have on the organization sponsoring the program.
C) some indication that the evaluation will really have an effect on the people who need the services of the program being evaluated.
D) the expected conflicts between the evaluator and the management that may well impact the quality of the project.

Question 8

An evaluation of high quality usually __________ than one of lower quality.

A) is based on data from a greater number of program participants
B) was designed with input from fewer stakeholder groups
C) is based on data from a more representative sample of program participants
D) employs less costly surveys and questionnaires

Question 9

The data collection is

A) best turned over to the program staff since they are on the scene and can easily handle the activity.
B) to be handled by someone without direct knowledge of the setting of the program since it is usually a rather mechanical process.
C) likely to be corrupted even by well-meaning program staff members if they are permitted to control data collection.
D) fairly simple since confidentiality is not an issue in applied social research projects.

Question 10

A written proposal outlining the steps to be followed in carrying out a planned program evaluation

A) should always be prepared.
B) can be omitted if the evaluation is conducted internally.
C) should always bind the evaluator to exactly the procedures described.
D) cannot realistically be prepared since so many unplanned events impinge upon program evaluations.

Question 11

Resistance to an evaluation is likely to be greatest when the evaluation is

A) a formative evaluation.
B) a cost-effectiveness evaluation.
C) an outcome evaluation.
D) a summative evaluation.

Question 12

Small improvements in the achievements of program participants *

A) can be very important if the program affects many people.
B) are seldom of interest to serious program managers.
C) cannot be justified in a politically-charged environment.
D) are often all that is expected by program staff members.

Question 13

Many members of the staff of human service programs resist program evaluations in their facilities because

A) evaluation methods cannot be sensitive to their concerns.
B) it may appear that they are losing control of their program.
C) their jobs on are on the line.
D) evaluators have a way of taking over.

Question 14

One way for evaluators to mollify the critics of a program evaluation project is

A) to demonstrate clear analytical skills.
B) to mold the project design to avoid any points of concern.
C) to listen to the worries of the critics and relieve all concerns that are founded on misperceptions of program evaluation.
D) for the evaluator to ally him/herself with the facility management, who will see to it that the correct procedures are followed.

Question 15

Goal-free evaluation refers to evaluations that are conducted

A) on programs with no goals.
B) when program managers cannot describe their goals.
C) when the program is designed to show that the government is "doing something" but no one expects improved outcomes.
D) when evaluators want to examine the program without their own expectations being affected by knowledge of the program's goals.

Question 16

One way to reduce fear and resistance to evaluation activities is to reassure staff that

A) the larger community will be better off knowing when a program is ineffective.
B) program participants (i.e. the clients) have a right to know how effective the program is.
C) documenting success would increase the organization's commitment to the program.
D) less that half of outcome evaluations result in the curtailment of a program.

Question 17

The social science model for program evaluation served to

A) get rid of a lot of ineffective services.
B) demonstrate that a non-significant statistical test showed that the program was ineffective.
C) introduce more creativity into program evaluation methods.
D) introduce additional rigor into program evaluation practices.

Question 18

Using expert opinion as a form of program evaluation is especially useful when *

A) the program is complex and there are few clear objective criteria of effectiveness.
B) there are readily observable outcomes from a program.
C) it is essential that the causal connection between the program and the outcome be definitely understood.
D) new medications are being evaluated.

Question 19

Finding discrepancies between client needs and services offered or between projected outcomes and achieved outcomes

A) shows why a service ought not be supported.
B) identifies ways to improve a program.
C) reveals the issues that must be faced in making program improvements.
D) shows that evaluators have done their work.

Question 20

Inappropriate reasons to evaluate a program include

A) documenting program activities to satisfy a funding agency.
B) demonstrating effectiveness prior to seeking a new source of funding.
C) seeking more efficient ways to provide social and educational services.
D) commissioning an evaluation to deflect criticism and postponing a decision.

Question 21

Considerable assistance can be obtained in planning a program evaluation from

A) published evaluations.
B) the Internet.
C) informal conversations with other evaluators.
D) all of the above

Question 22

An evaluation to focus on participants who have achieved the program goals

A) is usually a good basis for a summative evaluation.
B) is inherently dishonest.
C) lies at the heart of the social science model.
D) is called the Success Case Method.

Question 23

Describe how one would discuss random assignment to treatment or non-treatment conditions with a group of people who applied for a new oversubscribed job-training program.

Question 24

How would the difference between formative and summative evaluations affect the negotiations with program staff and other stakeholders in the planning of an evaluation?

Question 25

Discuss the advantages of involving the relevant stakeholders in the planning of an evaluation.

Question 26

If during the planning of an evaluation, an evaluator discovers that critical stakeholder groups differ in their views of central objectives of a program, what are the best courses of action for the evaluator?

Question 27

What can evaluators do to encourage stakeholders to take additional responsibility for programs in which they are involved?