Deck 11: Unobtrusive Measures

Question 1

In impact analysis, an experimental design is preferred to maximize internal validity.

Answer 1

True

Question 2

Impact analyses typically use quantitative data.

Answer 2

True

Question 3

Evaluation research is unlike traditional social science research because:

A) It is designed to test the implications of a social theory.
B) Program stakeholders have influence in how the study is designed.
C) Evaluation research cannot ethically use randomization.
D) Evaluation research rarely uses quantitative methods.
E) There is no dependent variable to be measured.

Answer 3

B

Question 4

The more complex the social problem, the more value that qualitative methods can add to the evaluation process.

Question 5

A scientific evaluation study can only measure one outcome.

Question 6

According to U.S. law, some sort of evaluation is required of all government programs.

Question 7

The investigation of how a social program works is called a mechanism evaluation.

Question 8

The direct product that a program delivers is the outcome.

Question 9

Evaluation research developed in tandem with government expansion during the Great Depression and World War II.

Question 10

Quasi-experimental designs are the preferred method for proving that a program created (or caused) a desired effect.

Question 11

Evaluation research began and developed in which time period?

A) 1910s-1920s
B) 1930s-1940s
C) 1950s-1960s
D) 1970s-1980s
E) 1990s-2000s

Question 12

Evaluability assessments generally rely on quantitative methods

Question 13

Stakeholders objectively define needs in a needs assessment.

Question 14

The type of evaluation research that determines if a new social program is needed or an old social program is still needed is called an impact assessment.

Question 15

With a black box evaluation study, understanding how a program works is not important.

Question 16

Researchers should always emphasize a strict social science orientation when conducting evaluation research.

Question 17

Process evaluation investigates how a service is delivered.

Question 18

If evaluation findings will be used to help shape and refine a social program, it is known as a formative evaluation.

Question 19

Developing supportive relations with stakeholder groups will increase the odds that the political process will not undermine the evaluation process.

Question 20

Evaluation research is conducted to investigate social programs.

Question 21

In Project New Hope, 677 low-income adults were offered jobs, childcare, and healthcare, while others were offered none of these things. In the end, only 27% of those who were offered jobs stuck with it long enough to lift themselves out of poverty. Levels of depression did not decrease, nor did self-esteem increase. However, the classroom performance of male the participants' male children increased. This example demonstrates the importance of:

A) Theory-driven evaluation
B) Process evaluation
C) Measuring multiple outcomes
D) Efficiency analysis
E) Stakeholders' perceptions

Question 22

The services delivered or new products produced by a social program are program:

A) Process
B) Outputs
C) Theory
D) Participation
E) Evaluation

Question 23

When evaluating their program, in which group exercise classes tended to increase several measures of participant health, researchers concluded that it didn't really matter how the group classes worked, just so long as it worked. This is an example of:

A) Black box theory
B) Program theory
C) Utilization theory
D) Theory-driven evaluation
E) Process theory

Question 24

The Government Performance and Results Act of 1993 requires what of government programs?

A) IRB oversight
B) Cost-benefit of analysis
C) Needs assessment
D) Evaluability assessment
E) Some sort of evaluation

Question 25

Evaluability assessments tend to rely on:

A) Experimental designs
B) Qualitative methods
C) Surveys
D) All of the above
E) None of the above

Question 26

Cost-benefit analyses and cost-effectiveness analyses are both what kind of evaluation research?

A) Needs assessment
B) Evaluability assessment
C) Process evaluation
D) Impact analysis
E) Efficiency analysis

Question 27

A study specifically designed to investigate whether a program can be evaluated is known as a:

A) Needs assessment
B) Evaluability assessment
C) Process evaluation
D) Impact analysis
E) Efficiency analysis

Question 28

Theory driven evaluation does which of the following?

A) Assesses the relative costs of a social program
B) Determines the generalizability of a program
C) Guides the investigation of a program process
D) Suggests which outcomes should be measured
E) Designates the most theoretically significant outcome

Question 29

The preferred method for maximizing internal validity in impact analysis is:

A) Process theory
B) Stakeholder orientation
C) Black box orientation
D) Experimental design
E) Qualitative methods

Question 30

If a program evaluation is written without regard to whether the program had the intended effect, but with very careful attention to how it will be received by other social science peers, it has what kind of orientation or approach?

A) Process
B) Stakeholder
C) Social science
D) Theoretical
E) Efficiency

Question 31

Evaluation research that investigates the process of service delivery is known as:

A) Needs assessment
B) Evaluability assessment
C) Process evaluation
D) Impact analysis
E) Efficiency analysis

Question 32

A community hires a researcher to determine whether a community center would be a benefit, in terms of reducing delinquency, promoting community service, and developing networks. To provide an answer, the researcher would conduct what sort of evaluation on community services?

A) Needs assessment
B) Evaluability assessment
C) Process evaluation
D) Impact analysis
E) Efficiency analysis

Question 33

One important difference between evaluation research and other social scientific research is:

A) Evaluation research is often guided by politics.
B) Evaluation research rarely uses qualitative research.
C) Evaluation research rarely uses quantitative research.
D) Evaluation research is conducted by professional researchers.
E) Evaluation research is not widely respected by academics.

Question 34

To find out if a new program is needed in a community, conduct a:

A) Process evaluation
B) Evaluability assessment
C) Impact analysis
D) Needs assessment
E) Efficiency assessment

Question 35

Impact analysis generally relies on what sort of research design?

A) Experimental and quasi-experimental
B) Open-ended surveys
C) Closed-ended surveys
D) One-shot case studies
E) Key informant interviewing

Question 36

Which of the following is not a typical input in social programs?

A) Clients
B) Customers
C) Cases
D) Participants
E)

Question 37

To lessen the potential detrimental impact of social programs, researchers should do all of the following, except:

A) Vary treatments among settings rather than among individuals within a setting
B) Use the minimum sample size required to adequately test the results
C) Minimize the number in the untreated control group
D) Test just parts of new programs, rather than the entire program
E) Compare the presence and absence of treatments, rather than those that vary in intensity

Question 38

The economic value of a social program when compared to the costs of that program is established in a:

A) Cost-benefit analysis
B) Cost-effectiveness analysis
C) Process analysis
D) Summative assessment
E) Needs assessment

Question 39

Which of the following is a typical reason why a program cannot be evaluated?

A) Management only wants its superior performance confirmed
B) Staff doesn't trust management to check on their performance
C) Personnel have no clear sense of what the program is trying to achieve
D) Program is not clearly distinct from other services provided by the agency
E) All of the above

Question 40

Who is eliminated from the research process in appreciative inquiry?

A) Stakeholders
B) Sponsors
C) IRB
D) Staff
E) Professional researchers

Question 41

What are the limitations of evaluation research when compared to other forms of social scientific research?

Question 42

What orientation to evaluation research expects researchers to be responsive primarily to the people involved with the program?

A) Stakeholder approach
B) Social science approach
C) Integrative approach
D) Academic approach
E) Client approach

Question 43

Name some of the steps listed in the chapter that can be taken when ethical standards can't be met.

Question 44

What is the difference between an evaluation research project that assumes a black box and one that seeks to develop a program theory? What are the advantages and disadvantages to each?

Question 45

Who are the individuals or groups that have some basis of concern about a social program?

A) Sponsors
B) Clients
C) Stakeholders
D) Researchers
E) Professors

Question 46

What is the difference between stakeholder and researcher orientation? Why does it matter in terms of results?

Question 47

Which of the following is not a question that evaluation projects focus on?

A) What is the level of need for the program?
B) Who can be eliminated from the program?
C) Can the program be evaluated?
D) What is the program's impact?
E) How efficient is the program?

Question 48

Outline the elements of a social program and how they are linked together, using the example of the DARE program detailed in the text.

Question 49

When is it appropriate to use quantitative methods for evaluation research?

A) For clear percentages and numerical scores
B) For quick comparisons of categories
C) For systematically tracking change
D) For comparing outcomes between an experimental and a control group
E) All of the above

Question 50

When the goal of evaluation is to compare alternative programs, it may be preferable to:

A) Randomize whole groups, not individuals, to different groups
B) Randomize individuals, not whole groups, to different groups
C) Randomly sample small groups from a small population
D) Make sure to have a small sample size
E) Only use this approach when studying small businesses and individuals

Question 51

How and why did evaluation research develop? How does it differ from other types of social science research?

Question 52

To evaluate a program designed to improve math scores for elementary school girls, girls were divided into treatment and control groups. Girls in the treatment group were required to attend an after-school program, while girls in the control group went home on time. Math exam scores were then compared between the two groups of girls. This type of program design is associated with:

A) Needs assessment
B) Evaluability assessment
C) Process evaluation
D) Impact analysis
E) Efficiency analysis

Question 53

Based on the information in Chapter 11, how would you evaluate the success of the DARE program? Explain your answer fully.

Question 54

Why is evaluation research more political than other forms of social research?

Question 55

Describe the special ethical considerations associated with evaluation research. How might a researcher lessen the detrimental impacts to subjects?

Question 56

When is it appropriate to use qualitative methods for evaluation research?

A) To uncover how different individuals react to treatment
B) For quick comparisons of categories
C) For systematically tracking change
D) For comparing outcomes between an experimental and a control group
E) None of the above

Question 57

Describe the potential problems for stakeholder orientation in evaluation research, in general and using the example of reducing welfare payments in favor of a workfare program.

Question 58

What steps can be taken to lessen the potential detrimental impact of social programs on participants? What other ethical issues should evaluation researchers consider, both in terms of social science ethics and federally mandated criteria?

Question 59

Explain the two types of efficiency analysis and provide examples where each would be helpful. Answer Key

Question 60

In what circumstances should qualitative methods be used in evaluation research? In what circumstances should quantitative methods be used? Use examples to demonstrate your answer.