Deck 16: Evaluation Research

Question 1

Briefly describe a research question that could be answered using evaluation research.

Answer 1

No Answer

Question 2

List three questions that a researcher should answer to determine the effectiveness of an evaluation research study.

Answer 2

No Answer

Question 3

Define needs assessment and describe the factors that a researcher should consider when completing this assessment.

Answer 3

No Answer

Question 4

Explain why it is important to identify the client population when developing a program evaluation

Question 5

Name three possible sources of potential problems to be addressed by an evaluation research study.

Question 6

Name three factors that evaluation researchers should consider when developing a solution to an identified problem.

Question 7

What is evaluability assessment? Name three things an evaluation researcher needs to know to conduct this type of assessment.

Question 8

Define goal specification. Explain why it is important for an evaluation researcher to consider both the importance of goals and their consequences.

Question 9

Distinguish between proximal and distal outcomes in evaluation research and give an example of each.

Question 10

What are social impact outcomes and what do they indicate?

Question 11

Name four groups who are common stakeholders in evaluation research and then identify their roles in this process. Why is it important to consider the different goals and expectations these stakeholders might have?

Question 12

What factors should a researcher consider when selecting the variables to be included in an evaluation research study?

Question 13

Name three advantages of conducting a theory-based program evaluation.

Question 14

What is a target population in evaluation research? How can a researcher ensure that the selected population is unbiased? What are the consequences of any potential bias?

Question 15

Distinguish between formative and summative evaluation. Give an example of each.

Question 16

Explain the importance of accurate program implementation in evaluation research. Describe two possible sources of implementation failure

Question 17

Explain how one can assess whether an evaluation research program was properly implemented.

Question 18

Define client resistance in evaluation research and explain why it might happen. How can this resistance be reduced?

Question 19

Define impact assessment in evaluation research. Name three criteria for evaluating impact.

Question 20

How is change assessed in evaluation research? Explain why it is important to assess change from both a statistical and a practical perspective.

Question 21

Describe how unintended effects can operate at the program implementation stage. Are those effects generally positive or negative?

Question 22

Describe the factors that should be considered in determining whether an evaluation program had the desired impact.

Question 23

How can degrees of change be evaluated in a program evaluation study?

Question 24

Is an effective program that is not continued a failure? Explain your reasoning.

Question 25

Describe a research methodology that can be used to address whether a program was effective.

Question 26

What is a dismantling design? How can it be used to evaluate whether a program was effective?

Question 27

What is a client and program variation design? What information does this design offer researchers about the program's effectiveness?

Question 28

Name and describe two research designs that can be used to assess whether an evaluation program can be improved.

Question 29

Explain why it is important to include both a pretest and a follow-up when assessing the program's effectiveness.

Question 30

What factors limit a researcher's ability to conduct a true experiment to assess the program's effectiveness?

Question 31

Describe two situations that allow an evaluation researcher to use random assignment to conditions, even if other situational factors cannot be controlled.

Question 32

Explain why the nonequivalent control group design is commonly used in evaluation research.

Question 33

Name two threats to internal validity that are unique to evaluation research.

Question 34

Define control group contamination. Describe three possible sources of this type of contamination.

Question 35

What is local history and when is it most likely to threaten the internal validity of an evaluation research study?

Question 36

What is a pre-experimental research design? Describe a situation where such a design can be useful to evaluation researchers.

Question 37

Name two research questions that program evaluators can answer using meta-analysis.

Question 38

Name three possible sources of null results for a program evaluation. How would conclusions about a program's effectiveness differ depending on the identified source of the null result?

Question 39

What is an efficiency analysis? Name one research question that can be answered and one question that cannot be answered by using such an analysis.

Question 40

Distinguish between instrumental, persuasive, and conceptual utilization of the information gained by a program evaluation.

Question 41

Name and define three criteria that increase the likelihood that the results of evaluation research will be used by decision makers.

Question 42

Explain how political motivations can affect whether the results of evaluation research are utilized or ignored.

Question 43

How are difference scores computed? Why are these scores considered unreliable by some statisticians? What factors can increase their reliability?

Question 44

What is the Reliable Change Index? Give one example of how it can be used in evaluation research.

Question 45

Evaluation research is primarily directed at evaluating the

A) validity of a theory.
B) effectiveness of a program or social intervention.
C) reliability and validity of a newly-developed measure.
D) the external validity of a research finding.

Question 46

Which of the following questions can be addressed by evaluation research?

A) To what extent did the program achieve its goals?
B) How can the program be improved?
C) How valid is the theory underlying the program?
D) all of the above

Question 47

The first step in an evaluation research project is

A) program monitoring.
B) impact assessment.
C) goal definition.
D) efficiency analysis.

Question 48

In evaluation research, which of the following determines whether a problem exists and then points to a solution?

A) needs assessment
B) impact evaluation
C) evaluability assessment
D) program monitoring

Question 49

In evaluation research, which of the following determines the information needed to conduct an evaluation?

A) needs assessment
B) impact evaluation
C) program monitoring
D) evaluability assessment

Question 50

A non-profit organization wants to design an effective AIDS prevention program for a developing country. They hire Dr. Joyeaux to assist them. He thinks they should first determine whether AIDS is indeed a problem in that country. Which program goal is he addressing?

A) impact evaluation
B) program monitoring
C) evaluability assessment
D) needs assessment

Question 51

A non-profit organization wants to evaluate the effectiveness of their AIDS prevention program. They hire Dr. Joyeaux to assist them and he raises the concern that the program is being delivered to the wrong people. Which program goal is he addressing?

A) identifying potential causes
B) identifying the client population
C) evaluability assessment
D) needs assessment

Question 52

The client population of a program consists of

A) everyone living in the geographic area served by the program.
B) the intended recipients of the program's services.
C) the people who make use of the program's services.
D) decision-making and policy-setting officials at the agencies that fund the program.

Question 53

Identifying the client population of a program is important because

A) a program's services can be offered to people who do not actually need it.
B) in the context of designing interventions, the client population is the problem to be addressed.
C) it is the only way to identify an appropriate theory.
D) the characteristics of the client population drive the entire evaluation processes.

Question 54

The best program evaluations

A) are theory based.
B) use high-utility dependent variables.
C) address policy-relevant variables.
D) do all of the above

Question 55

Needs assessment is an important part of program evaluation. Which statement about this process is TRUE?

A) The client population need not be identified at this stage in the evaluation process.
B) Because program evaluation has an applied focus, theories are not relevant to needs assessment.
C) Policy makers' perceptions that a problem exists might be inaccurate.
D) The goals of an intervention are determined prior to the involvement of the program evaluators.

Question 56

To conduct a proper evaluation, researchers must know, among other things,

A) the goals of the program.
B) the expected consequences of achieving the program's goals.
C) the size of the effect the program is expected to have.
D) all of the above.

Question 57

Which of the following is NOT necessary for conducting a proper program evaluation?

A) identifying the program goals
B) designing a program that will have the desired consequences.
C) identifying a theory that links the elements of the program to the expected outcomes
D) the ability to randomly assign members of the client population to treatment and control conditions

Question 58

In evaluation research, the process of goal specification is analogous to the process of ________ in experimental research.

A) developing manipulations of independent variables
B) operationally defining dependent variables
C) identifying extraneous variables that need to be controlled
D) identifying the research population

Question 59

Which of the following can negatively affect the goal specification process of evaluation research?

A) Different stakeholder groups might have different ideas about what a particular goal means in practice.
B) Program administrators might state program goals in vague terms.
C) It is difficult to choose which behaviors best indicate goal attainment.
D) all of the above.

Question 60

__________ are the people who have a vested interest in the program being evaluated.

A) Evaluators
B) Stakeholders
C) Clients
D) Impact assessors

Question 61

The stakeholders in a program being evaluated include

A) opinion leaders in the community where the program is implemented.
B) the agencies that fund the program.
C) program clients.
D) all of the above.

Question 62

Which of the following group members would NOT be considered stakeholders in a program?

A) the people who designed the program
B) opinion leaders in the community where the program is being implemented
C) the researchers conducting the evaluation
D) policy makers who decide whether the program should continue to be funded.

Question 63

Which of the following is generally NOT an outcome assessed in evaluation research?

A) emotions: how clients should feel after participating in the program
B) knowledge: what clients should know after participating in the program
C) behavior: what clients should do after participating in the program
D) attitudes: what clients should believe after participating in the program

Question 64

As part of the goal specification process, evaluation researchers should specify all of the following EXCEPT

A) when in the program the expected effects should appear.
B) the size of the expected effects.
C) the expected cost of administering the program.
D) how long the program's effects should last.

Question 65

The goal specification process includes all of the following EXCEPT

A) operationally defining the program's goals.
B) prioritizing the program's goals.
C) identifying both proximal and distal outcomes.
D) identifying the value of the program's goals relative to their expected cost.

Question 66

________ outcomes are direct effects of the program that can be expected to occur while the clients are taking part in the program.

A) Social impact
B) Proximal
C) Distal
D) Client-centered

Question 67

________ outcomes are indirect effects of the program that occur after clients have completed the program.

A) Stakeholder driven
B) Proximal
C) Social impact
D) Policy-focused

Question 68

Distal outcomes are also called

A) social impact outcomes.
B) proximal causes.
C) policy-focused outcomes.
D) stakeholder goals.

Question 69

Dr. Kuh has designed a program to help first generation college students succeed in their first year. A proximal outcome would be

A) midterm grades for the students' first fall semester.
B) four-year graduation rates.
C) courses chosen for the sophomore year.
D) scores on the Graduate Record Exam.

Question 70

Dr. Kuh has designed a program to help first generation college students succeed in their first semester. A distal outcome would be

A) first-term midterm grades of a C or better.
B) higher graduation rates.
C) improved first-semester class attendance.
D) all of the above

Question 71

________ outcomes occur in clients' broader social environment and are often reflected in social indicators.

A) Social impact
B) Proximal
C) Policy-focused
D) Client-centered

Question 72

Distal program outcomes are

A) under the direct control of the program.
B) indicators of the program's ecological validity.
C) unrelated to proximal outcomes.
D) usually the least important of a program's outcomes.

Question 73

Which of the following statements about the role theory plays in program evaluation is TRUE?

A) Because programs are applications of psychology, theory has no role to play in them.
B) Theories can be consulted at the project design stage, but they play no role in the implementation stage.
C) The cost of conducting theory-based evaluations usually exceed the benefits of doing so.
D) If a program is based on a theory, the evaluation can function as a test of that theory.

Question 74

Including program theory in the evaluation of a program

A) helps evaluators assess whether the conditions necessary for program success are met.
B) allows researchers to identify the most important program components.
C) provides evaluators with possible reasons for why a program did not meet its objectives.
D) does all of the above.

Question 75

A ________ evaluation involves continuously assessing how well a program is being implemented while it is being carried out.

A) summative
B) cumulative
C) formative
D) client-focused

Question 76

A ________ evaluation addresses the overall effectiveness of a program relative to its goals.

A) summative
B) cumulative
C) formative
D) process

Question 77

A formative evaluation should include procedures to assess

A) whether the program is being implemented properly.
B) sources of client resistance to the program.
C) any unintended effects of the program.
D) all of the above.

Question 78

Summative evaluations are _________; formative evaluations are __________.

A) essential; nonessential
B) high stakes; low stakes
C) atheoretical; theory-based
D) cost-effective; expensive

Question 79

Dr. Bhalla wants to assess whether on-the-job training improves employee retention. Which of the following would be a summative assessment of the program's effectiveness?

A) anonymously assessing employee satisfaction at the midpoint of the program
B) scores on a standardized achievement test given at the end of the program
C) assessing pre-training ability levels
D) all of the above

Question 80

Dr. Chou is conducting a program evaluation. Midway through, he realizes the program is not being implemented as it was designed. Dr. Chou's most likely based this assessment on

A) a formative evaluation.
B) a survey of stakeholders.
C) a summative evaluation.
D) the theory he used to design the program.