Deck 10: Quasi-Experimental Designs and Program Evaluation

Question 1

When it comes to control over the conditions in the experiment or over the assignment of participants to groups in the experiment,a researcher in a natural setting is likely to have

A) less control than a researcher in a laboratory setting.
B) more control than a researcher in a laboratory setting.
C) the same amount of control as a researcher in a laboratory setting.
D) more control over the conditions in the experiment but less control over assignment of participants.

Answer 1

less control than a researcher in a laboratory setting.

Question 2

A high school teacher conducted a test of a new approach to teaching math.Students were given a pretest when their math class began and a posttest at the end of the semester.The students' math performance improved.The teacher learned near the end of the semester,however,that in their science classes the students were using new computer software that included much of the math the teacher covered in his course.Which of the following threats to internal validity does the new computer software represent?

A) selection
B) regression
C) history
D) testing

Answer 2

history

Question 3

At a large university,a group of education specialists tested the effectiveness of a new academic improvement course.Students seeking help at the university counseling center because of academic difficulties were asked to participate in this 4-week program.Only students who were judged to be deficient in reading comprehension and other study-related skills were chosen for the program.Students who sought help at the counseling center for emotional difficulties were included in the study as a comparison group.These students received the usual treatment offered at the counseling center.A group of 30 students completed the academic-improvement program at the counseling center.Average test grades from the students' courses for the two groups were compared before (midterm exams)and after (final exams)the program.Analyses revealed that a statistically significant majority of the students were doing better in school after completing the academic-improvement program than before.
What type of research design was used for this study?

Answer 3

A nonequivalent control group design was used.

Question 4

When individuals' performance on a posttest differs from their initial testing not because of a treatment but because of familiarity with the measure,a __________ threat to internal validity is likely.

A) testing
B) instrumentation
C) regression
D) contamination

Question 5

The most critical defining characteristic of a true experiment is often seen to be a high degree of control over the

A) arrangement of experimental conditions.
B) random assignment of participants to experimental conditions.
C) choice of dependent variables.
D) systematic manipulation of independent variables.

Question 6

In clinical trials involving tests of new medical treatments,it may be extremely difficult to get patients to agree to be randomly assigned to either the treatment or the control group.In these situations researchers can use

A) natural groups designs.
B) quasi-experimental designs.
C) yoked control designs.
D) matched groups designs.

Question 7

How does a time series design with nonequivalent control group differ from a nonequivalent control group design? Identify a threat to internal validity that is controlled by adding a nonequivalent control group to a simple interrupted time series design.

Question 8

Why is the nonequivalent control group design superior to the one-group pretest-posttest design in terms of controlling for threats to internal validity?

Question 9

Distinguish between evaluation of outcome and evaluation of efficiency as these are used in program evaluation.

Question 10

Lab-based experiments are likely to have greater _________,and experiments in natural settings are likely to have greater __________.

A) social application; theoretical implications
B) internal validity; external validity
C) consequences for more people; control
D) all of these

Question 11

At a large university,a group of education specialists tested the effectiveness of a new academic improvement course.Students seeking help at the university counseling center because of academic difficulties were asked to participate in this 4-week program.Only students who were judged to be deficient in reading comprehension and other study-related skills were chosen for the program.Students who sought help at the counseling center for emotional difficulties were included in the study as a comparison group.These students received the usual treatment offered at the counseling center.A group of 30 students completed the academic-improvement program at the counseling center.Average test grades from the students' courses for the two groups were compared before (midterm exams)and after (final exams)the program.Analyses revealed that a statistically significant majority of the students were doing better in school after completing the academic-improvement program than before.
What major threats to internal validity are controlled by this research design?

Question 12

One factor that may make it difficult to interpret the result of even a true experiment in a natural setting is contamination (when there is communication of information about the experiment between groups of participants).Briefly describe the three types of problems that can result from contamination.

Question 13

Research done in natural settings,as compared to research done in laboratory settings,is more likely to emphasize

A) abstract goals.
B) methodological goals.
C) ethical goals.
D) practical goals.

Question 14

At a large university,a group of education specialists tested the effectiveness of a new academic improvement course.Students seeking help at the university counseling center because of academic difficulties were asked to participate in this 4-week program.Only students who were judged to be deficient in reading comprehension and other study-related skills were chosen for the program.Students who sought help at the counseling center for emotional difficulties were included in the study as a comparison group.These students received the usual treatment offered at the counseling center.A group of 30 students completed the academic-improvement program at the counseling center.Average test grades from the students' courses for the two groups were compared before (midterm exams)and after (final exams)the program.Analyses revealed that a statistically significant majority of the students were doing better in school after completing the academic-improvement program than before.
Identify and describe two threats to internal validity in this study.That is,show why there are at least two plausible alternative hypotheses for the obtained change in grades other than due to the academic-improvement program.

Question 15

Experiments in natural settings are likely to differ from laboratory experiments on four critical dimensions: goals,control,external validity,and consequences.Briefly describe the nature of the difference for each of these four dimensions.

Question 16

In a study of the effectiveness of a treatment for depression,a psychologist assessed patients' symptoms of depression using a reliable questionnaire both before and for several months after treatment.The results indicated that the patients experienced a decrease in their symptoms over the 8-month time period of the study.One threat to internal validity the researcher should consider is

A) maturation.
B) regression.
C) testing.
D) all of these

Question 17

In the context of conducting experiments in natural settings,random assignment of participants to conditions is

A) usually perceived by possible participants as the fairest procedure.
B) the fairest procedure if the effectiveness of the treatment is known.
C) the fairest procedure if the effectiveness of the treatment is not known.
D) usually perceived by those in positions of authority as the fairest procedure.

Question 18

A physician wants to test the effectiveness of a new allergy medication.The physician consults a researcher who regularly conducts clinical trials.The physician tells the researcher that she has a backlog of 100 patients she is treating effectively with the current leading medication,but who could potentially benefit even more from the new medication.The researcher recommends that the physician do a true experiment because of the availability of

A) a waiting-list control group.
B) a quasi-treatment control group.
C) an excess-demand control group.
D) a convenience control group.

Question 19

The difference between a true experiment conducted in a natural setting and a quasi-experiment conducted in a natural setting is that the

A) quasi-experiment will have greater external validity.
B) quasi-experiment will have greater internal validity.
C) quasi-experiment will lack a comparison condition.
D) none of these

Question 20

The external validity of research done in natural settings is likely to be emphasized more when the research represents

A) an experiment done to address a specific question raised by a specific company.
B) an extension of a specific laboratory finding.
C) social experimentation as the basis for large-scale changes.
D) a theoretically motivated social psychology experiment.

Question 21

One of the main ways that true experiments differ from quasi-experiments is that true experiments use

A) correlational methods.
B) random selection from the population.
C) random assignment to conditions.
D) all of these

Question 22

Students on two college campuses serve as treatment and control groups in a study investigating the effectiveness of an alcohol-abuse prevention campaign.A well-known student on one of the campuses dies of alcohol intoxication in the course of the study; students on the other campus did not learn of the student's death.The reaction of other students to the student's death on their campus could represent a potential threat to the internal validity of the study called

A) history.
B) selection.
C) additive effects of selection and history.
D) additive effects of selection and maturation.

Question 23

A researcher trains observers to complete checklists while observing children's behavior on the schoolyard during recess.Over the course of the study,observers become more reliable in their observations.Any effect of a treatment in this study might be confounded with an ____________ threat to internal validity.

A) observation
B) instrumentation
C) additive
D) expectancy effect

Question 24

When,from the outset of an experiment,differences exist between the kinds of individuals in one group and those in another group,there is a potential threat to internal validity called

A) contamination.
B) attrition.
C) selection.
D) additive effect of selection and history.

Question 25

One threat to the internal validity of research that affects both true experiments and quasi-experiments is

A) selection threats.
B) additive effects with selection.
C) experimenter expectancy effects.
D) all of these

Question 26

Participants for a treatment group are chosen because they score very low on a pretest measure of performance.When their performance improves on the posttest,the researcher

A) should consider the possibility that statistical regression influenced posttest scores.
B) can rule out testing and instrumentation threats to internal validity.
C) can be confident that the treatment was effective.
D) must conclude that additive effects with selection are responsible for the outcome.

Question 27

A psychologist examines the effect of a new therapy by first assessing patients' symptoms using a pretest,then administering eight weeks of therapy,and then administering a posttest.Based on this research design,the psychologists will be able to

A) make a causal claim about whether the treatment is effective.
B) make a claim about the applicability of the treatment to other potential patients.
C) both (A) and (B)
D) neither (A) nor (B)

Question 28

Contamination that results from communication of information between groups of participants in an experiment can lead to all of the following except

A) resentment on the part of participants receiving less desirable treatments.
B) greater compliance with the instructions within each group.
C) rivalry among participants receiving different treatments.
D) general diffusion of treatments across the groups.

Question 29

The hallmark of a quasi-experiment is

A) high levels of control over treatment conditions.
B) lack of random assignment of participants to conditions.
C) absence of any specific intervention or treatment.
D) random assignment of participants to conditions.

Question 30

When events that occur during the course of a study have a different effect on one group of participants than on another,the possible threat to internal validity is

A) differential regression.
B) selection.
C) history.
D) additive effects of selection and history.

Question 31

A threat to internal validity that occurs when information about the experiment is communicated between the different groups of participants is known as

A) expectancy effects.
B) contamination.
C) novelty effects.
D) the Hawthorne effect.

Question 32

An intervention in an office setting leads employees to be pleased that the management is interested in their welfare.If the employees' performance improves in this situation,the researcher should be concerned about potential

A) novelty effects such as the Hawthorne effect.
B) lack of discontinuity in the time series.
C) Campbell effects.
D) contamination effects.

Question 33

An argument for the internal validity of quasi-experiments

A) cannot be made because of inherent threats to internal validity in these designs.
B) can always be made solely on the basis of the results of the quasi-experiments.
C) can only be made if a corresponding true experiment has been done.
D) can be made on the basis of supplemental data and logical analysis in addition to the results of the quasi-experiment itself.

Question 34

Random assignment to conditions is used in true experiments to control which of the following threats to internal validity?

A) selection
B) testing
C) history
D) subject attrition

Question 35

The Sports Illustrated jinx refers to the idea that athletes who are featured on the cover of the magazine after exceptional performances typically perform worse in subsequent competitions.The most likely statistical explanation of this phenomenon is

A) regression toward the mean level of performance.
B) an increase standard deviation for the amount of pressure the athletes experience.
C) an increase in the mean level of athletes' arrogance.
D) a decrease in the standard deviation of opposing athletes' effort during subsequence competitions.

Question 36

The one-group pretest-posttest design is

A) a bad experiment.
B) an important quasi-experimental design.
C) the best design for clinical research.
D) least affected by threats to internal validity.

Question 37

A student used a nonequivalent control group design to examine the effectiveness of a video application the library uses to introduce first year students to the library's resources.Which potential threat to internal validity would she examine by comparing the pretest scores for both groups?

A) a maturation threat
B) a selection threat
C) a regression threat
D) an instrumentation threat

Question 38

Which of the following threats to internal validity is controlled in the nonequivalent control group design?

A) testing.
B) additive effect of selection and testing.
C) additive effect of selection and history.
D) differential regression.

Question 39

When people's performance changes because they are enthusiastic or energized by an intervention,the results of a study are likely affected by

A) diffusion of treatment.
B) treatment enthusiasm.
C) contamination.
D) novelty effects.

Question 40

The main problem associated with subject attrition during the course of a study is that

A) regression toward the mean on the dependent variable measure is likely.
B) the group with the most participants remaining at the end of a study will likely have higher posttest scores.
C) the researcher can no longer use statistical analysis to understand the results.
D) groups initially created to be equivalent may no longer be equivalent.

Question 41

When individuals in one group of a nonequivalent control group design experience natural changes at a faster rate than individuals in the other group,there is danger of a threat to internal validity called

A) additive effects of selection and history.
B) additive effects of selection and maturation.
C) additive effects of selection and instrumentation.
D) differential statistical regression.

Question 42

When two groups in a nonequivalent control group design do not differ on the pretest,the researcher can conclude that

A) the two groups are equivalent.
B) the two groups are not equivalent on any dependent variables.
C) the two groups will show the same natural growth rate over the course of the study.
D) the two groups are equivalent only on the dependent variable measured on the pretest.

Question 43

After implementing a program to improve admitting procedures in a hospital emergency room,a researcher seeks to determine whether the program is being implemented as proposed.Which research method will best serve this purpose?

A) observational method
B) survey method
C) experimental design
D) quasi-experimental design

Question 44

The effect of a treatment in a simple interrupted time-series design is best indicated by

A) an increasing trend in the dependent variable that is present both before and after the treatment.
B) a decreasing trend in the dependent variable that is present both before and after the treatment.
C) a clear discontinuity (abrupt increase or decrease) in the dependent variable at the point the treatment is administered.
D) a gradual change in the dependent variable that begins just as the treatment is implemented.

Question 45

A psychologist tests the effect of an incentive program (i.e.,positive reinforcement for desired behavior)in a residential treatment facility for delinquent youth.He randomly assigns one building of the large facility to receive the treatment.Residents in a second building serve as a control group.During the course of the one-month study,an event happens in the treatment group that forces full lockdown of the building for one week.The threat to internal validity the psychologist must consider is

A) selection.
B) history.
C) additive effect of selection and history.
D) contamination.

Question 46

A study using a simple interrupted time-series design examined people's responses to the September 11,2001 terrorist attacks using a questionnaire measure of religion and faith.Archival data were available for the population four months before and after the attacks.The researchers observed that people's religious feelings and faith increased in the months following the attacks.One threat to internal validity that should be considered in this study is

A) the September 11, 2001 attacks.
B) regression toward the mean level of religious feeling.
C) that religious feeling increased, but other measures such as honesty and curiosity did not.
D) seasonal variation in people's feelings of religion and faith.

Question 47

Researchers analyzed records of hospitalization for heart disease in two Ohio cities,Bowling Green and Kent.A city ordinance enacted in Bowling Green banned smoking in public areas.Results indicated that hospitalization rates for the two cities were similar before the smoking ban,but hospitalization rates decreased in Bowling Green compared to Kent following the smoking ban.A potential threat to internal validity that will need to be ruled out is

A) the amount of information about heart disease presented in national media.
B) changes in the state sales tax on cigarettes.
C) changes in doctors' focus on symptoms of heart disease in Bowling Green (because of publicity for the ban), but not in Kent.
D) all of these

Question 48

The relationship between basic research and applied research can best be described as

A) an independent relationship.
B) a reciprocal (or circular) relationship.
C) a hierarchical relationship with basic research taking precedence over applied research.
D) a top-down relationship with applied research taking precedence over basic research.

Question 49

Even when pretest scores are the same,on average,for treatment and comparison groups in a nonequivalent control group design,the two groups may not be equivalent because

A) the pretest measure is unlikely to be relevant to the dependent variable.
B) the posttest measure is unlikely to be the same as the pretest measure, and the two groups must be equivalent at the posttest.
C) the fact that the two groups are equal on the pretest measure does not ensure that the groups are equivalent on other characteristics relevant to the study.
D) the natural growth rates of two groups from different populations are likely to be the same, but the pretest estimate of equality may still be in error.

Question 50

Program evaluation is a hybrid discipline whose professional activity

A) relies almost exclusively on the experimental method.
B) is confined almost entirely to the discipline of psychology.
C) has as its basic goal providing feedback to providers of human service activities.
D) focuses primarily on conducting basic research.

Question 51

Program evaluation represents perhaps the extreme case of

A) basic research.
B) applied research.
C) theory development.
D) experimental control.

Question 52

A clinical psychologist examined the effect of a treatment designed to reduce the amount of cigarette smoking in the day room of an inpatient psychiatric facility.His treatment consisted of a several components to maximize its potential effectiveness (e.g.,increasing the number of available activities; giving patients responsibility for care of the unit's plants,fish,etc.; the opportunity to exchange cigarettes for positive reinforcers).At the end of three months,his data indicated that cigarette smoking had declined significantly.Based on this,he is able to conclude that

A) any of the components, implemented by itself, would reduce smoking in the day room.
B) the treatment will work only when all of the components are present.
C) neither (A) nor (B)
D) both (A) and (B)

Question 53

The major threat to internal validity in the simple interrupted time series design is

A) history.
B) maturation.
C) selection.
D) additive effect of selection and history.

Question 54

If the residents,staff,and facilities of a nursing home where a quasi-experiment was done are likely to be different from those in other nursing homes,the ______________ of the findings may be questioned.

A) external validity
B) statistical significance
C) internal validity
D) novelty

Question 55

Social policy research (such as the research done by Ellen Langer on mindfulness and the effects of living in a nursing home)serves the purpose(s)of

A) informing policy decisions only.
B) advancing scientific theory and knowledge only.
C) advancing scientific theory and knowledge and informing policy decisions.
D) advancing scientific theory and knowledge or informing policy decisions but not both.

Question 56

Which of the following is not one of the four questions addressed in program evaluation?

A) assessment of needs
B) evaluation of process
C) evaluation of outcome
D) decision making efficiency

Question 57

A human services agency was interested in finding out whether establishing a food distribution center in their community would provide a service the community would use.Which of the following types of assessments in program evaluation would this agency find useful for this question?

A) evaluation of process
B) assessment of needs
C) evaluation of outcome
D) evaluation of efficiency

Question 58

Among the following research designs,which allows researchers to rule out the most threats to internal validity?

A) one group pretest-posttest design
B) nonequivalent control group design
C) simple interrupted time series design
D) times series design with nonequivalent control group

Question 59

Strong evidence of a treatment effect in a time series with nonequivalent control group design is indicated by

A) discontinuity in the comparison group but not in the treatment group.
B) discontinuity in the treatment group but not in the comparison group.
C) discontinuity in both the treatment group and the comparison group.
D) discontinuity in neither the treatment group nor the comparison group.

Question 60

A researcher was asked to evaluate the outcome of a program implemented to enhance recycling in a community.Which method will the researcher likely select to answer the question of whether the program was effective?

A) participant observation
B) random groups design
C) cross-sectional survey design
D) time series with nonequivalent control group design

Question 61

Even when controlled,scientific research studies established the effectiveness of a home healthcare program for Medicare recipients,decision makers eliminated the program.The best explanation for this decision lies in

A) politicians' unwillingness to fund scientific research.
B) the political and social realities of complex problems such as healthcare.
C) the public's greater acceptance of basic research over applied research.
D) none of these

Question 62

Which of the following is likely to be the most important benefit from the use of program evaluation based on sound experimental methodology?

A) advancing the development of basic research in the social sciences by focusing on social policy
B) advancing the influence and prestige of the social sciences in decision making among social policy makers
C) providing information to policy makers that can be helpful in making more informed choices among possible treatments for social problems issues
D) providing definitive information to policy makers such that program evaluation will eventually become the primary basis of social policy decisions

Question 63

Perhaps the greatest difference between basic research and program evaluation lies in the differences in

A) the political and social realities surrounding each of them.
B) statistical procedures.
C) the importance of the research.
D) methodology.