Deck 10: Impact Program Evaluation and Hypothesis Testing

Question 1

McDavid, Huse, and Hawthorne (2013) identify key features that are important in distinguishing impact program evaluation for performance measurement: Which of the following is NOT included in that list?

A) Frequency
B) Issue
C) reporting of results
D) Attribution of outcomes

Answer 1

C

Question 2

An impact program evaluation can demonstrate that a human service program includes all EXCEPT which of the following answers:

A) implemented with a poor program design.
B) successful.
C) unsuccessful due to theory failure.
D) unsuccessful due to program failure.

Answer 2

A

Question 3

A program hypothesis does NOT do which of the following statements:

A) makes explicit the assumptions underlining a human service program outcome (result).
B) establishes a framework to bring internal consistency to the implementation of a human service program.
C) enables inputs, process, outputs (including quality outputs), and outcomes to be examined for internal consistency.
D) if followed by the correct program design, will result in the desired result.

Answer 3

D

Question 4

The purpose of comparison in impact program evaluation is to determine all of the following EXCEPT:

A) what actually happens to clients as a result of client's participation in the program.
B) which part of the program caused the client improvement.
C) do program clients show improvement.
D) how do we know for sure that the improvement is actually due to the program and not some external factor(s).

Question 5

When two groups are compared in impact program evaluation, one of the following is NOT correct:

A) one group serves as the experimental group and the other serves as the comparison group.
B) a single group serves as the experimental group and also serves as the comparison group.
C) all groups are tested at the end of the program and the scores added and divided by the number of participants.
D) a counterfactual is evidence of what would have happened to participants if they were not in the program.

Question 6

Which of the following is NOT a type of impact program design ?

A) The single group pretest/posttest design
B) The nonequivalent comparison group design
C) The randomized experimental design
D) The single group posttest design

Question 7

Which of the graphically depicted designs from your text is NOT correct?

A) R 01 x 02
B) 01 02 x
C) 01 x 02
D) R 03 04

Question 8

In impact program evaluation, the Treasury Board of Canada (2010) identified 7 major threats to internal validity. Which of the following is INCORRECT

A) History
B) Selection bias
C) Using an experimental group
D) Testing

Question 9

The nonequivalent comparison group design begins with the issue of threats to internal validity. Which of the following is NOT correct.

A) Two groups are tested at two different time periods.
B) The experimental group does not serve as its own comparison group.
C) Two distinct groups are created: one group becomes the experimental group where individuals will participate in the program and the other is a group is individuals who are "statistically similar to the clients participating in the program, but who will not participate in the program."
D) Persons who are eligible to participate in the program but cannot be served because of inadequate funding.

Question 10

In examining threats to internal validity history, selection bias and testing are described below. Which of the following is NOT a description of those threats?

A) Events outside the program differentially affect the scores of the experimental or comparison group.
B) Differences in the dropout rate between the experimental group and the comparison group differentially affect effect the scores between the two groups.
C) Members of the experimental and comparison groups are not truly equal on all important factors.
D) The pretest differentially affected the learning of the experimental or comparison group.

Question 11

The measurement or observation is related to the client characteristics. Which of the following does NOT describe a client characteristic:

A) behavior
B) knowledge
C) time in the program
D) condition

Question 12

An impact program evaluation is a hypothesis-testing activity. Using a domestic violence example, which of the following statements is NOT correct?

A) Then a woman would successfully leave her abusive partner
B) If we could identify the major barriers
C) If we could successfully eliminate or reduce those barriers with a well-designed and well-implemented human service program
D) Then we should see a positive change in participant's lives and an increase in self-sufficiency

Question 13

If a hypothesis has been tested, which of thefollowing is NOT true?

A) The program was implemented as designed.
B) The desired results are achieved.
C) The hypothesis is supported.
D) The hypothesis is not supported unless the participant is not self-sufficient.

Question 14

In an Impact Program Evaluation, which of the following statements is NOT true?

A) Research designs vary in complexity.
B) Research designs vary in length.
C) Research designs vary in timeliness.
D) Research designs vary in feasibility.

Question 15

An initial measurement in a single group pre-test/posttest design that includes the initial measurement of observation is called a pretest and takes place:

A) before the program.
B) during the implementation of the program.
C) a result at the end of the program.
D) at any point during implementation of the program.

Question 16

There are several features of impact program: evaluation and performance measurement. Both share some features and some features are different. Which of the features listed below is NOT correct for performance measurement?

A) Ongoing
B) Attributes of outcomes are assumed
C) Episodic
D) Designed to address more specific performance issues that are constant over time

Question 17

There are several features of impact program: evaluation and performance measurement. Both share some features and some features are different. Which of the following is NOT correct for impact program evaluation?

A) Attributes of outcomes are to be determined.
B) Attributes of outcomes are to be determined.
C) Designed to address specific stakeholder questions at a particular point in time.
D) Episodic.

Question 18

Theory failure describes a situation in which a human service program is implemented according to design, but the anticipated outcome is not achieved. Which of the features listed below does NOT describe theory failure?

A) Comes about because of a flaw in the hypothesis underlying the program
B) Certain causal processes will lead directly to the desired outcome
C) The hypothesis is not supported when the program is implemented according to the program design, but the result is not the desired outcome
D) It is designed to address specific stakeholder questions

Question 19

A human services program director would do an impact program evaluation when there is a need to do all EXCEPT which of the following:

A) compare two groups to determine if there is a difference between the level at which program clients show improvement.
B) if there can be only one point of evaluation.
C) to determine that the improvement is due to the program and not some external feature.
D) what happens to clients who participate in the program versus those who do not.

Question 20

An analogy that describes how random assignment works would include all EXCEPT which of the components:

A) people standing in a line.
B) a coin is tossed in the air.
C) every other person in the line is assigned to the comparison group.
D) if the coin shows tails, the person is assigned to the experimental group.

Question 21

Impact program evaluation differs from other types of evaluation in that the focus is on changes in program participants and not on changes in organizations and communities.

Question 22

An impact program evaluation seeks to establish a cause-and-effect relationship between a program and its outcome (result).

Question 23

When a human service services program fails to achieve its desired outcomes (result) due to theory failure, the results are invalid for any type of program evaluation.

Question 24

The essence of impact program evaluation is comparison.

Question 25

The pretest and posttest design in a single group pretest/posttest design does not need to be created. It may be any test, observation, or measurement:

Question 26

To be able to say with more assurance that the program caused the change, we must identify threats to internal validity.

Question 27

The strongest impact research design and the one that deals most strongly with threats to internal validity is the single group pretest/posttest design.

Question 28

Ethical issues such as withholding participation in a program for experimental purposes and legal requirements such as withholding participation if a client meets eligibility requirements often cause human service administrators to rely on less religious designs when conducting impact program evaluations.

Question 29

Random assignment to groups is the least powerful in controlling for selection bias.

Question 30

Statistically similar means that the characteristics of the individuals that comprise the comparison group are the same or similar to the characteristics of the individuals that comprise the experimental group on all relevant characteristics.

Question 31

Impact program evaluation's intent is to answer the question: Does the program actually have an impact and does it achieve positive outcomes (results) for _____________.

Question 32

Impact program evaluation attempts to demonstrate that an outcome (result) is ______________ to the program and not some other variable.

Question 33

Using a logic model terminology, a successful program is one where there is ___________ between the program hypothesis (theory) and implementation of the program design (cause) that produces the desired outcomes (result).

Question 34

When a program fails to achieve the desired outcome (result) due to ________________ the program has not been properly tested and we cannot make inferences about the success or failure of the program.

Question 35

Comparisons in impact program evaluation are usually made in one of two ways: two different groups are compared or one group is compared to ___________.

Question 36

Define theory success, theory failure. and program failure, describe the differences between them and give an example of each within a human service program.

Question 37

Describe the two ways comparison in impact program evaluation is achieved and give an example of each.

Question 38

Impact program evaluation attempts to determine if a human service program actually works. In order to make a cause-and-effect statement a counterfactual has to be created that would identify what would have happened to individuals had they not participated in the program?
You are the program director of a program that works with young women to improve their parenting skills and you are charged to evaluate the program to determine if the program has made a difference in knowledge and behaviors of the women who participated in the program. Describe 1) the program hypothesis, 2) which of the two ways that you will use to compare two groups, 3) create the cause-and-effect statement and, if necessary, create a counterfactual that identifies what would happen if participants would not have participated in the program, and 4) describe the desired result that should be the outcome of the program and how you would determine if it had been achieved.