Deck 9: Evaluation Research

Question 1

Researchers are attempting to conduct an intervention providing public transportation vouchers to residents as a method of reducing neighborhood crime rates.In hindsight,researchers discovered that participants in the treatment group were already less likely to have a criminal record than the general population to which they were compared.This problem is an example of

A)selection bias.
B)matching procedure.
C)treatment condition.
D)control condition.

Answer 1

A

Question 2

Why are randomized field experiments considered the "gold standard" of evaluation research?

A)They usually take place in a laboratory,where conditions can be closely controlled.
B)They tend to have a higher rate of participation than other research studies.
C)They are not biased by the differences between treatment and control groups.
D)They are relatively easy to administer compared to other sociological research methods.

Answer 2

C

Question 3

When using a matching procedure to study the impacts of class size on student achievement,researchers would most likely assemble a control group by matching participants based on which characteristic?

A)participation in extracurricular activities
B)student height
C)parental marriage status
D)prior school performance

Answer 3

D

Question 4

When researchers adjust for the effects of additional variables to ensure the treatment and control groups are equivalent,they are using

A)statistical controls.
B)nonequivalent comparison design.
C)pre-post design.
D)matching procedure.

Question 5

What is the term for when treatment and control groups are formed by a procedure other than randomization?

A)external validity
B)randomized field experiment
C)selection bias
D)nonequivalent comparison design

Question 6

An intervention hopes to reduce teenage texting while driving by introducing an intervention in a local driver's education course.Researchers will measure the young people's self-reported propensity to text and drive before and after the intervention.What is one issue confronting the pre-post design of this study?

A)Researchers would achieve more accurate validity with reflexive controls.
B)Statistical controls might better account for pre- and post-design outcomes.
C)Young people's awareness of the dangers of texting and driving might increase throughout a driver's education course even without the intervention.
D)Because of selection bias,young people who were already aware of the dangers of texting and driving may be more likely to enroll in the study.

Question 7

What defines quasi-experimental research methods?

A)They lack random assignment of individuals to treatment and control groups.
B)They avoid comparison of treatment and control groups.
C)They tend to have very small sample sizes.
D)Researchers who conduct these studies are trained in biological sciences.

Question 8

What is one reason why a study investigating whether a new urban housing assistance program reduces homelessness might not achieve internal validity?

A)At the same time of the study,developers flood the market with affordable apartments,making housing more accessible to homeless populations.
B)While the intervention may work in the city,urban participants might differ from homeless populations in a rural area.
C)Researchers received money from a foundation aimed to end homelessness,which might have a vested interest in the results.
D)Homeless populations are difficult to follow since they tend to move around so much.

Question 9

One of the main goals of evaluation research is translation,which means what?

A)the process of implementing the components of an evaluation research project on a larger scale
B)attempts by qualitative sociologists to fill the gaps in quantitative studies
C)the strategy of making internationally based research applicable to a domestic,U.S.-based audience
D)basing a research project on diverse subject populations who may speak different languages

Question 10

In the ROWE (Results-Only Work Environment)intervention,researchers conducted qualitative fieldwork to get a sense of major work-life issues facing Best Buy employees.This information was then used to write relevant survey questions.This strategy of using qualitative information to inform quantitative research is called

A)time-series design.
B)formative research.
C)reflexive controls.
D)matching procedure.

Question 11

What is one method through which researchers can focus their projects on areas of greatest concern?

A)developing their research questions in consultation with key stakeholders
B)basing their questions on well-known academic theories
C)conducting research in the most densely populated cities
D)prioritizing the needs of the research projects' funders

Question 12

What is the distinction between a time-series design and multiple time-series design?

A)Each strategy attempts to understand how participants conceive of time,but multiple time-series considers how conceptions of time may vary across cultures.
B)Time-series design takes measurements throughout the course of a single day,while multiple time-series design takes measurements across a series of weeks or months.
C)Both strategies take multiple measures of outcomes over time,but multiple time-series design compares effects over multiple locations.
D)Both strategies take multiple measures of outcomes over time,but multiple time-series design gathers measurements at multiple points prior to the intervention and once afterward.

Question 13

In an evaluation study of the impact of car breathalyzers on drunk driving,researchers collected both quantitative survey data and qualitative interviews.In this case,the quantitative evaluation could determine

A)the efficacy of their survey instrument.
B)why the intervention worked.
C)whether the intervention worked.
D)how individuals experienced the intervention.

Question 14

In pre-post design,researchers measure the outcome of interest once before the intervention,introduce the intervention,and then measure the outcome again.Why is this considered the simplest method of reflexive control?

A)Measurements rarely vary across pre-post design.
B)Participants serve as their own controls.
C)They effectively measure outcomes in long-term evaluation studies.
D)Few variables are likely to change on their own regardless of intervention.

Question 15

In the ROWE intervention,Best Buy's work teams were split into two categories.One category of work teams implemented a Results-Oriented Work Environment,in which employees could work whenever and wherever they wanted as long as they got their work done,while the second category continued work as usual.How would sociologists classify the latter category?

A)experimental group
B)control group
C)treatment group
D)randomized group

Question 16

Urban poverty researchers have developed an intervention aimed at reducing predatory lending (i.e. ,unfair financial lending practices,often targeting poor communities of color).Which of the following research methods is best suited to measure whether the intervention worked?

A)quantitative records of lending practices based on organizational data
B)qualitative interviews with financial lenders
C)ethnographic observations of neighborhood banks and lending agencies
D)time-use diaries of financial lenders

Question 17

Which of the following projects best illustrates the logistical challenges of evaluation research?

A)Researchers may want to apply the findings of their study on Uber driving in New York City to five additional cities across the United States.
B)A study of reduced-cost remedial college classes measures student progress by sorting program applicants into a treatment group,which would receive reduced-cost remedial college classes,and a control group,which would be enrolled in full-priced traditional college courses.
C)In an intervention in factory labor,researchers found 60% of jobs could be downsized or replaced by robots to save corporate profits.Local labor unions might challenge and object to the study's results.
D)To study an intervention in Uber driving,researchers must be able to locate large numbers of Uber drivers,who are often spread out throughout a city,and involve them in research.

Question 18

Self-reported measures (such as surveys)are more sensitive to participants' moods and to social desirability bias,which means what?

A)study participants' desires to present themselves in a positive light
B)researchers' inclination to study populations with whom they personally identify
C)the ability to measure how well someone gets along with others
D)individuals' inclination to participate in research that provides financial compensation

Question 19

Why might it be unreasonable to ask whether an urban housing program would eliminate all homelessness?

A)Homelessness is too politically charged to involve in an intervention study.
B)Evaluation projects should focus broadly on what makes a program successful.
C)The issue of homelessness impacts rural communities,not simply urban ones.
D)Most interventions are designed to produce important but modest effects.

Question 20

One purpose of the Results-Only Work Environment intervention study at Best Buy was to

A)increase employee wages.
B)reduce employees' control over their schedules.
C)reduce work-family conflict.
D)enhance employee fluency in workplace computer software.

Question 21

Randomized field experiments,according to the text,are often considered the "gold standard" of evaluation research.However,that doesn't mean they are perfect.In 2-4 sentences,explain what a randomized field experiment is and appraise some of its strengths and weaknesses.

Question 22

List and briefly describe the three major challenges of evaluation research.Choose one type of challenge and hypothesize in 1-2 sentences how that challenge might manifest itself in a study of an intervention reducing class sizes in an elementary school.

Question 23

When researchers attempted to re-create the STAR study in California,results were less positive than anticipated.According to the text,what is one possible explanation for these different outcomes?

A)Reducing class sizes for all K-3 classrooms in California required an enormous decrease in teachers.
B)The new teachers hired for the intervention were vastly overqualified.
C)California class sizes were likely too small to produce desired results.
D)By Tennessee's standards,California's reduced class sizes were not actually "small."

Question 24

A study attempted to intervene in heroin overdoses in the Midwest by giving away an expensive antidote medicine to local clinics and users' support centers.However,the intervention achieved only small effects,with treatment areas having only slightly lower reported overdoses than control areas.A cost-benefit analysis of such a study might determine what?

A)The intervention is too expensive to warrant its expansion.
B)The high costs are worth achieving even small effects in a life-or-death social problem.
C)Translating the results to a larger environment will be easy.
D)The difficulty of estimating the value of life negates the purpose of a cost-benefit analysis.

Question 25

In the Student/Teacher Achievement Ratio (STAR)study,why did the researchers create two separate intervention groups?

A)The sample size was too large to only accommodate one intervention group.
B)The study could also evaluate the effectiveness of a cheaper intervention-teacher's aides.
C)The central evaluation question was whether larger class sizes improved student learning.
D)Since the cost of the project was so low,researchers evaluated multiple interventions.

Question 26

A group of labor researchers studying fast food workers hope to measure the effect of eliminating on-call scheduling (when workers are scheduled to be "on-call" but may or may not have to come into work)on worker health and well-being.One unanticipated consequence researchers found in the treatment group was that even though formal on-call schedules were eliminated,fast food managers continued to call in workers on their days off,a phenomenon which has been documented in similar studies.Based on what we know from the STAR study,what lesson do evaluation researchers need to keep in mind?

A)It is important to rely on specific scientific knowledge that has been established in prior evaluation research.
B)Adequate resources must be in place before launching a small-scale social intervention.
C)Intervention studies should avoid implementing programs incrementally.
D)Researchers should look to self-reported measures to conclude whether the program worked.

Question 27

What action did researchers in the Tennessee (STAR)experiment undertake to alleviate ethical concerns?

A)Students in the control group were placed in larger class sizes than existed prior to the study.
B)Students in the control group experienced the same size of classes that existed prior to the study.
C)Students in the control group were financially compensated for their participation in the study.
D)Students in the treatment group were financially compensated for their participation in the study.

Question 28

Which sentence best describes the role politics play in intervention research?

A)Politics often play a supportive role in evaluation research.
B)What is important to policy makers never changes as new officials are elected.
C)Policy makers may lose interest in a program before the evaluation is complete.
D)Researchers should bemoan the fact that evaluation research takes place in a political context.

Question 29

An intervention into children's test scores measures the impact of children who regularly study for two hours per night.However,some children in the intervention fail to consistently study for two hours per night,affecting the results.What type of issue is this?

A)logistical
B)political
C)ethical
D)numerical

Question 30

In a 2002 report,researchers found that in reduced-size classes in California,34% of third-grade students scored above the national median in reading,compared with what percentage of third graders in regular-size classes?

A)22
B)32
C)42
D)52

Question 31

A poverty organization in Los Angeles is testing the intervention of cash handouts to women on welfare.Though they may produce robust sociological data,randomized field experiments such as this one carry serious ethical concerns.What is one likely ethical concern of such a project?

A)The intervention may not work.
B)If cash handouts for one group of women will increase,cash handouts for another group will decrease from their previous levels.
C)The study participants may spend their money too quickly.
D)Researchers may be arbitrarily denying some people of a known benefit.

Question 32

Which evaluation study is considered one of the MOST important educational investigations ever undertaken?

A)the STAR study
B)the Homebase program
C)the ROWE study
D)the Abstinence Education Program

Question 33

In medical studies,what is the term meaning the treatment the participants would normally receive outside of the study and not some lesser treatment?

A)standard treatment
B)control group
C)treatment group
D)randomized sample

Question 34

Compare and contrast in 1-2 sentences the concepts of "reasonable scope" and "answerability" in evaluation research.Consider the research question proposed by the Results-Only Work Environment (ROWE)study and explain in 2-3 sentences the extent to which it is reasonable and answerable.

Question 35

Though all students in the STAR study benefited from smaller classes,the gains were larger for minority students in both the first and second grades.Why is this finding important?

A)It proves that minority students are underachievers.
B)The findings will likely not be adopted in majority white school districts.
C)White students actually performed better with larger class sizes.
D)Researchers want to reduce achievement gaps among various social groups.

Question 36

In an intervention study of aging populations,a Midwest department of health attempted to measure the impact of providing in-home care to elderly patients.One group of patients,who had previously been treated in a hospital,was returned to their homes and visited by nurses and volunteers several times per week,while another group stayed in a traditional hospital setting.In 2-4 sentences,explain which group in this scenario constitutes the treatment group and which constitutes the control group.

Question 37

A study of white collar crime (e.g. ,financially motivated crimes perpetrated by white collar professionals)attempted to measure the intervention of teaching Wall Street financiers about the social ramifications of white collar crime.The intervention did not prove successful.What will likely be the next steps for the study?

A)Other cities will want to adopt the program.
B)The program will not be expanded.
C)Additional funding will be sought.
D)Stakeholders will be consulted before expansion.

Question 38

Sociologists at the University of Pennsylvania wish to conduct an intervention involving providing free breakfast and lunch for all students under 18 and its impacts on educational success.They will randomly assign state school districts to the intervention and will match the treatment and control sample based on population characteristics.What is one reason why this study,like all randomized field experiments,might be difficult to implement?

A)Randomized field experiments have low internal validity.
B)Randomized field experiments are often high cost.
C)Policy makers will likely want to take their time making a decision on the intervention.
D)Youth education is not one of the highest priority social problems.

Question 39

At what school level was the 1996 Abstinence Education Program implemented?

A)public high schools and universities
B)private high schools and the higher grades of middle schools
C)public high schools,middle schools,and the higher grades of elementary schools
D)public preschools and elementary schools,with parental permission

Question 40

Health-care researchers in Texas are conducting a randomized field experiment on the effects of providing free health care to people over 65 years old.One of the logistical issues with this study,however,is that people in the control group tended to drop out of the study at higher rates since they were not benefiting from the treatment.What do sociologists call this phenomenon?

A)selection bias
B)reflexive control
C)differential attrition rates
D)social desirability bias

Question 41

If you were to help plan an intervention study measuring the effects of parents' reading to their children before bedtime on students' test scores,propose and explain in 1-3 sentences one step you might take to avoid biased results.

Question 42

In 1-3 sentences,propose an intervention aimed at alleviating a major social problem where you live.Then,list at least one ethical,one logistical,and one political concern researchers may encounter in carrying out the study.

Question 43

Explain the concept of a cost-benefit analysis,and compare and contrast how at least two constituents might respond to a cost-benefit analysis of the Results-Only Work Environment intervention.Possible constituents may include,but need not be limited to,the company (Best Buy)or its employees.

Question 44

The textbook looks in depth at two examples of evaluation research: the Tennessee Class-Size Experiment (or the Student/Teacher Achievement Ratio,STAR),and the Abstinence Education Program.Choose one of these programs and explain in 3-4 sentences the focus and outcome of the study as well as at least one difficulty in conducting the study or translating the study's results.

Question 45

Food studies scholars are attempting to measure the interventions of access to farmers' markets in neighborhoods that qualify as "food deserts" (areas with little access to grocery stores with fresh produce).Researchers plan to look at quantitative outcomes regarding health and well-being,but what advantages might qualitative data also provide? Explain your answer in 2-3 sentences.