Deck 10: Quasi-Experiments and Evaluation Research

Question 1

In terms of internal validity (i.e., least vulnerable to internal validity threats is first, most vulnerable is last), experimental methods, correlational methods, and quasi-experimental methods would be organized as:

A) experimental > quasi-experimental = correlational
B) experimental = quasi-experimental > correlational
C) experimental = correlational > quasi-experimental
D) experimental > quasi-experimental > correlational

Answer 1

D

Question 2

Program evaluation research differs from strictly experimental research in which of the following ways? (circle all that apply)

A) program evaluators are exclusively involved in planning and implementing the intervention or treatment program
B) program evaluation is unique in that it almost always utilizes quasi-experimental designs
C) evaluation research has an explicitly political character
D) program evaluation research will often build some flexibility into the research design, so that program stakeholders can potentially make adjustments and improvements to the treatment or intervention as new information becomes available

Answer 2

C,D

Question 3

What are the main goals of "program efficacy" studies? (circle all that apply)

A) determine whether the effects of a planned intervention occur as planned
B) determine if the planned intervention can be delivered effectively, given existing restraints (i.e., can it be executed in the real world under normal, expected circumstances)
C) test the intervention under the "best of all possible worlds" to provide the best chance of treatment effectiveness
D) determine the prevalence and severity of a social problem, and accordingly, whether a program is necessary

Answer 3

A,C

Question 4

This type of evaluation is used to determine whether a planned program can be implemented and delivered as expected, given existing restraints.

A) program feasibility studies
B) program development studies
C) program efficacy studies
D) cost-benefit analyses

Question 5

A nonprofit organization wanted to implement its new, revolutionary program for addressing homelessness. As part of the series of evaluation studies that assessed this planned program, evaluators conducted a pilot study that resulted in several suggestions for revising the general procedures for introducing the program to homeless populations, changing some of the wording of questionnaires, and adding another treatment condition to provide greater insight into the nuances of the treatment effects. This type of evaluation study would be considered:

A) program feasibility evaluation
B) needs assessment evaluation
C) summative evaluation
D) formative evaluation

Question 6

What conditions must be met to be able to appropriately conduct an impact (or outcome) evaluation? (circle all that apply)

A) the individual elements of a program must be defined sufficiently for evaluators to determine whether the program is present or absent in a given situation
B) there must be some basis for observing or otherwise assessing final outcomes in the presence or absence of the treatment program
C) there must be some indication that the treatment was administered as planned, and that it was delivered as planned to the appropriate population
D) the goals of the program must be explicit enough for evaluators to develop operational definitions of those goals (i.e., outcome measures that will assess whether these goals are met)

Question 7

The reliability coefficient for a measure can reflect: (circle all that apply)

A) the extent that observed scores at Time 1 and Time 2 can be attributed to true score differences rather than random error
B) the correlation between Group 1's set of scores on a measure and Group 2's set of scores on the same measure
C) the correlation between Time 1 and Time 2 scores for the same people on the same measure
D) the (un)likelihood that observed differences between Time 1 and Time 2 scores are due to a regression artifact

Question 8

Which of the following is true regarding regression toward the mean?

A) when two sets of scores are imperfectly correlated, high scores in the first set will inevitably be lower in the second set
B) it is an inevitable consequence of examining the association between scores on two imperfectly related measures
C) it is a particular problem whenever at least one set of scores is selected for its extremity.
D) A and B
E) B and C
F) all of the above

Question 9

Which of the following techniques is an accepted method of controlling for regression artifacts?

A) comparing observed posttest scores with a predicted posttest score
B) matching posttest scores in the treatment group to scores in a separate "control group," after assignment to treatments has been determined
C) comparing outcomes for participants in a treatment group to those from a control group using a score that reflects the pattern of responses on several covariates
D) A and B
E) A and C
F) all of the above

Question 10

With propensity scores analyses, the difference between observed and counterfactual scores reflects changes attributable to:

A) regression toward the mean
B) true score changes, absent of treatment effects
C) treatment effects
D) preexisting differences between comparison groups

Question 11

Propensity scoring is often preferred when using a matching design for all of the following reasons, EXCEPT:

A) propensity score matching controls for extraneous variables even if they are not measured
B) the assumptions about the data are more relaxed for propensity score analyses
C) propensity scores control for several covariates simultaneously
D) matching is not determined after assignment to treatment conditions has been determined

Question 12

Not all social programs are amenable to basic comparison group designs, particularly if they involve an entire population. One alternative design that assesses the effect of a quasi-experimental treatment is a time series design. Researchers used a time series design to assess the impact of a new tax on fatty foods. They concluded that prior to the tax implementation on January 1, fatty food consumption was very high; after the tax went into effect, fatty food consumption declined considerably. Data were collected every four months. Before their conclusion can be accepted, what other factors are particular concerns for time series designs that must be taken into account? (circle all that apply)

A) data observed at one time point tend to be less related to adjacent time points than other points in the series
B) data observed at one time point tend to be more related to adjacent time points than other points in the series
C) there tend to be systematic cycles of change (e.g., with the types of food people eat) that would lead consumption to rise and fall, independent of any treatment or tax implementation
D) if data are nonlinear, it makes time series analyses extremely hard to interpret

Question 13

Which of the following graphs reflect possible outcomes of a growth curve modeling design? (circle all that apply)

A)
B)
C)
D)

Question 14

There are many types of longitudinal or quasi-experimental designs that assess data patterns over time. _____(a)_____ are concerned with differences in change between those who fall directly below a cutoff point to those who fall directly above the cutoff in terms of an outcome; _____(b)_____ are concerned with the shape or pattern of the data across multiple time points.

A) growth curve modeling designs; interrupted time series designs
B) regression discontinuity designs; growth curve modeling designs
C) interrupted time series designs; regression discontinuity designs
D) comparison time-series designs; growth curve modeling designs

Question 15

The "Stroop" Task is an example of:

A) a measure of retrieval
B) a measure of memory
C) a measure of perception
D) a measure of attention
E) none of the above

Question 16

If a participant's eye gaze is directed toward one particular stimulus rather than other stimuli involved in the study, this is likely an indicator of:

A) greater interest in the stimulus
B) social approval of the stimulus
C) a stimulus that is difficult to process or understand
D) a negative stimulus
E) all of the above

Question 17

Research that uses recall tasks:

A) is better suited for the study of memory errors
B) has found order effects in recall ability for different items
C) is most often interested in the accuracy of items recalled
D) is most often interested in the number of items recalled
E) tends to focus on "intrusions" and the reasons for these errors

Question 18

Having participants complete a series of sentence unscrambling tasks containing words meant to induce a certain mood is an example of:

A) subliminal priming
B) supraliminal priming
C) concept priming
D) parafoveal processing
E) sequential priming

Question 19

The underlying logic of _________ tasks is that positive primes should facilitate the evaluation of positively viewed target stimuli, while positive primes should inhibit judgment of negatively viewed stimuli.

A) subliminal priming
B) sequential priming
C) automatic evaluation
D) lexical decision
E) pronunciation

Question 20

Which of the following cognitive processes or components is NOT involved in the processing experienced by a participant completing the Implicit Association Test (IAT)?

A) response interference
B) processing speed
C) cognitive busyness
D) automatic associations

Question 21

This form of psychophysiological measurement is concerned with understanding the relationship between oxygen flow in the brain and various processes and emotional states; use and interpretation of this measure requires a deep understanding of the roles and responsibilities of the different brain regions.

A) facial electromyography
B) Functional magnetic resonance imaging (fMRI)
C) Event-Related potentials (ERPs)
D) Electroencephalography (EEG)

Question 22

Describe the basic and defining features of a quasi-experiment, and provide one specific example situation where using a quasi-experimental design would be preferred.

Question 23

The nation's obesity epidemic is declining! Or so two researchers claimed after analyzing their longitudinal data on the inherited aspects of people's self-reported weight. Specifically, they found that the heaviest 10 percent of the country tend to have slightly less heavy children. Therefore, if we extrapolate this effect out to three or four generations, even the heaviest people will much lighter than their great great grandparents, and considerably closer to the nation's average (healthy) weight. (a) Is this conclusion warranted - why or why not? (b) What other explanations might there be for this observed decrease in people's weight? (c) Explain whether this is a regression artifact or a regression fallacy.

Question 24

The correlation between Time 1 and Time 2 assessments of the same measure was r = 0.90. (a) Explain what this correlation has to do with reliability. (b) What information does this number tell us about participants' "true scores" on that measure? (c) How much variation in scores can be attributed to true score differences, and explain why you can draw this conclusion.

Question 25

In City A, researchers investigated high school students' alcohol use patterns after the police captain announced a new task force charged with curbing underage drinking. The researchers wanted to determine whether underage drinking actually decreased after this announcement, and realized that City B, a city that was quite similar to City A, had not made any such announcement. If the researchers were to use a comparison time-series design, what potential problems would they be avoiding by not using a simple time-series design. In other words, (a) what additional information does a comparison time-series design provide over a simple time series design? (b) Furthermore, provide two examples for the variables and data that would be included in a comparison time-series design, with one example comparing City A and City B, and the other example only using City A.

Question 26

Evaluators were interested in the long-term efficacy of a new early intervention education programs for Lilliputians, by far the most underprivileged population in the world. There had been considerable past research that showed without intervention, academic achievement in Lilliputians actually gets worse over time, whereas achievement in privileged populations tends to increase. For example, in a previous cross-sectional assessment of achievement, the average academic achievement for these populations is provided in the table below. To investigate the effect of this program on Lilliputians' subsequent academic achievement, the researchers decided to give themselves the best possible chance of detecting a treatment effect and compare the achievement of the highest performing Lilliputians to students from the privileged population with similar scores on the pretest (i.e., the lowest performers from the privileged population). Their results can be seen in the figure below. They conclude that the educational program was a bust, as it appears that even the best Lilliputians showed no change in their academic aptitude, while those in the privileged population, who were not exposed to any treatment, improved considerably. Was their conclusion warranted? In your response, (a) explain whether their conclusion was warranted or not; (b) explain how regression toward the mean might be affecting the results for both populations; (c) explain whether you think the education program was successful - in other words, based on what you know about regression toward the mean, discuss the likely outcomes for both populations in the absence of any treatment, and then compare that to the observed results when a treatment was present for the Lilliputians. (d) Finally, if you had all of the resources available in the world, provide an example for how you could use a regression discontinuity design to test for the causal effects of an educational intervention on the worst-performing Lilliputians. Be sure to describe the time frame of the study and how often data points would be collected, how participants would be assigned to groups, and after the intervention is applied, how you would determine whether it had an effect.

Question 27

What are some of the most prominent forms of program evaluation? How do they differ in terms of the goals of the evaluation?

Question 28

What are some situations in which matching may be necessary? What are some potential pitfalls of using matching techniques?

Question 29

What kinds of research questions or contexts might lend themselves to the use of growth curve modeling; of the regression-discontinuity design?

Question 30

Why are propensity scores any better than post-hoc matching? Isn't a propensity score just an aggregate of many matched characteristics? Would we obtain the same results if we used post-hoc matching across 10 variables and if we created a propensity score for those 10 variables - why or why not? Would pairing propensity scores that are not exactly equal be effectively the same as pairing two participants that are matched on nine out of 10 variables?

Question 31

You are interested in how teenagers think about decision making, and want to use naturalistic observation to assess the frequency of different thoughts, and valence of those thoughts, that a person experiences when asked to respond to a given stimuli (e.g., when asked whether he or she would try a new type of illicit drug). How could you do it? Could you use facial expressions in some way? What about eye movement or other unobtrusive physiological movements or responses? How else could you observe cognition (albeit indirectly) as unobtrusively and as "naturally" as possible? What might your coding scheme look like for these behavioral observations?