Deck 14: Designing Experiments

Question 1

To reduce bias in experimental results, all of these are useful except for which one?

A) Assigning participants to each treatment using an alternating procedure.
B) Assigning participants to each treatment using random numbers.
C) Blinding participants to which treatment they are in.
D) Including control treatments.

Answer 1

A

Question 2

Which of these is not a good method to reduce sampling error in experimental results?

A) Making sure that non-treatment factors were balanced in different groups.
B) Organizing subjects into blocks with treatments varying between blocks.
C) Organizing subjects into blocks with treatments varying within blocks.
D) The use of large sample sizes.

Answer 2

B

Question 3

An experimental unit is an individual or a group of individuals who have been assigned to an experimental treatment ___________ other individuals or groups.

A) different from
B) independently of
C) randomly to
D) separately from

Answer 3

B

Question 4

What is a control group?

A) A group for which the null hypothesis is true.
B) A group for which we know all the parameter values.
C) A set of experimental units that do not receive the interesting treatment but are otherwise similar.
D) A set of experimental units that receive the interesting treatment but under more controlled conditions.

Question 5

When testing a new drug for an illness, which of these is a major reason to use a control group?

A) Control groups eliminate all other factors that may cause statistical noise.
B) More than one group is required to perform a statistical test for differences between a response and an expected response.
C) People often get better just from thinking they have been treated, regardless of the actual effectiveness of the drug.
D) Statistical power is maximized in tests that use two groups compared to those that just use one.

Question 6

When testing a new drug for an illness, which of these is not a major reason to use a control group?

A) It allows us to distinguish the effects of different drugs from one another.
B) It allows us to distinguish the effects of the stress of being treated from the effects of the drug.
C) It allows us to distinguish the effects of typical improvement over time from the effects of the drug.
D) It allows us to distinguish the placebo effect from the effects of the drug.

Question 7

If the experiment involves giving drugs, we provide a ____ to the control group, whereas if the experiment involves surgical procedures, we provide a ____ operation to the control group.

A) blank : fake
B) fake : blank
C) placebo : sham
D) sham : placebo

Question 8

In experiments, when only the study subjects don't know which treatment group they are in, we call this a ____ - blind study, whereas if both the subjects and the people administering the treatment don't know, we call this a ____ - blind study.

A) individual : pairwise
B) simple : complex
C) single : double
D) solo : group

Question 9

Why is blinding important when designing studies?

A) Researchers may deliberately manipulate results if they know the groups.
B) Researchers may accidentally treat genuine and placebo individuals differently.
C) Researchers must not be able to know any details about the study subjects.
D) Researchers must treat all treatments the exact same way.

Question 10

Increasing the sample size reduces the magnitude, or risk, of all of the following except which one?

A) Bias
B) Noise
C) Sampling error
D) Type II error

Question 11

True replication is more than just the absolute number of experimental units; the units considered must be ______ as well.

A) Independent
B) Measured
C) Separate
D) Variable

Question 12

If the DNA sequences of a bunch of individuals from a family with a history of a specific genetic disease are compared to DNA sequences in a data base, what limits the power of tests to discover the causes of the disease?

A) The family members aren't independent, reducing the effective sample size.
B) The family members may have different sequences from the individuals in the database.
C) The individuals in the database are from different families, reducing the effective sample size.
D) The individuals in the database may have different sequences from the individuals in the family.

Question 13

When treatments in a study have different sample sizes, we say it is which of the following?

A) Clumped
B) Dispersed
C) Mixed
D) Unbalanced

Question 14

Experiments with two treatments per block are _____ designs, and experiments with more than two treatments per block are _____ designs.

A) single : multiple
B) single : variational
C) paired : randomized block
D) paired : unpaired block

Question 15

Why is it important to be cautious when judging the results of an experiment comparing extremely different treatments to one another?

A) The effect may not scale linearly, so smaller treatment differences may have no effect.
B) The effect observed may be not replicable if the experiment is performed again.
C) The effect seen may not be due to the extreme treatment used but due to some other factor instead.
D) The effect when doing an extreme experiment will always be unrealistic.

Question 16

Consider the figure shown indicating the results on an experiment growing a marine invertebrate in four different environments based on values of two factors. Two different temperatures and two different salinities were used. Which of the following is correct?
​
​

A) The effect of low versus high temperature depends on the salinity.
B) The effect of low versus high salinity depends on the temperature.
C) There is no interaction.
D) The two factors interact with one another.

Question 17

Consider the figure shown indicating the results on an experiment growing a marine invertebrate in four different environments based on values of two factors. Two different temperatures and two different salinities were used. Which of the following is incorrect?
​
​

A) The effect of low versus high temperature depends on the salinity.
B) The effect of low versus high salinity depends on the temperature.
C) There is no interaction.
D) The two factors interact with one another.

Question 18

Which of the following aspects of good experimental design is the one that observational studies are unable to provide?

A) Balancing
B) Controls
C) Extreme treatments
D) Randomization

Question 19

What are the primary benefits of matching when doing observational studies?

A) Matching eliminates bias and sampling error.
B) Matching eliminates noise and sampling error.
C) Matching reduces bias and sampling error.
D) Matching reduces noise and sampling error.

Question 20

Consider estimating the 95% confidence interval for the difference between two population means with a desired margin of error of 5 cm. What would the approximate (i.e., using the 2SE rule of thumb) minimum sample size be for each of the samples if the population standard deviation is 15 cm?

A) n = 60
B) n = 72
C) n = 84
D) n = 96

Question 21

Imagine you are estimating the difference between two population means by comparing 95% confidence intervals. If you use the 2SE rule of thumb and have desired margin of error of 5 cm, what is the approximate minimum sample size for each of the samples if the population standard deviation is 15 cm?

A) n = 108
B) n = 120
C) n = 132
D) n = 144

Question 22

For a given population variance, what is the relationship between sample size and expected margin of error?

A) There is a linear pattern (i.e., constant improvement) where larger sample sizes give larger margins of error.
B) There is a linear pattern (i.e., constant improvement) where larger sample sizes give smaller margins of error.
C) There is a nonlinear pattern (i.e., diminishing improvement) where larger sample sizes give larger margins of error.
D) There is a nonlinear pattern (i.e., diminishing improvement) where larger sample sizes give smaller margins of error.

Question 23

If testing for a 4 cm or more difference in means between two populations using a two-sample t-test with a = 0.05, what would the approximate (i.e., using a desired power of 0.80) minimum sample size be for each of the samples if the population standard deviation is 16 cm?

A) n = 256
B) n = 266
C) n = 276
D) n = 286

Question 24

Consider a situation in which you are interested in whether there is a 3 cm or larger difference between the means of two populations that each have a population standard deviation is 15 cm. You decide that a power of 0.080 is appropriate for your purposes. If you use a two-sample t-test with a = 0.05, what would the approximate minimum sample size be for each of the samples?

A) n = 380
B) n = 390
C) n = 400
D) n = 410

Question 25

Which performing 15 statistical tests, what is the Bonferroni corrected a* value that should be used instead of a = 0.05?

A) a* = 0.03
B) a* = 0.015
C) a* = 0.0033
D) a* = 0.0015

Question 26

Which of the following is the main problem arising when performing many statistical tests?

A) After the first test, the accuracy of each additional test decreases.
B) After the first test, the precision of each additional test decreases.
C) The overall risk of Type I error adds up.
D) The overall risk of Type II error adds up.

Question 27

An experimental unit is an individual or group of individuals who have been assigned an experimental treatment independently of other individuals or groups.

Question 28

A clinical trial is a study testing the effects of a new drug, operation, or other medical intervention.

Question 29

To reduce bias, all study subjects should be arranged according to a parameter of interest, and then an alternating procedure (e.g., A, B, A, B, etc.) should be used to assign the participants to the groups.

Question 30

The way that blocks are used in experiments is to assign individuals in a balanced manner to the blocks and then randomly assign each block to one of the treatments.

Question 31

Control groups in clinical trials are always expected to show no change in the variable of interest over time.

Question 32

A well-designed study will have only one control group.

Question 33

Randomization should never be carried out by listing subjects and then choosing experimental units for each treatment by alternating back and forth along the sequence.

Question 34

Assigning experimental units to each treatment alphabetically is bad practice.

Question 35

Experiments with humans should be double-blind, but experiments with animals don't require blinding.

Question 36

Reducing noise by making experimental conditions constant is always a good practice when designing experiments.

Question 37

For a given total sample size, a balanced design will result in the highest statistical power.

Question 38

Randomized block design uses three or more treatments.

Question 39

When factors interact, that is because we see effects when factors are combined that are not predictable from when each factor is tested in isolation.

Question 40

We use matching in observational studies to reduce, but not eliminate, bias and sampling error.

Question 41

With matching, every individual in the treatment group is paired with a control individual having the same or closely similar value of the parameter of interest.

Question 42

Estimating the required sample size for an experiment is often done to safeguard against undue harm to the study population.

Question 43

Estimating the required sample size for an experiment is often done for practical economic reasons.

Question 44

Doubling the sample size will tend to double the power of a statistical analysis.

Question 45

If we do four statistical tests in situations where the null hypothesis is true and use an a = 0.05 threshold, there is an 18.5% chance we have made at least one Type I error.

Question 46

If we do four statistical tests in situations where the null hypothesis is true and use an a = 0.05 threshold, there is a 21.5% chance we have made at least one Type I error.

Question 47

Using the Bonferroni correction reduces our overall risk of Type I error but increases the risk of Type II error for factors that have minor and moderate effects.

Question 48

Explain what blinding and double-blinding are and why they are important for clinical trials. Are they also needed for animal studies?

Question 49

Compare and contrast "balancing" and "blocking" with respect to experimental design. What do they both do, and in which ways do they differ?

Question 50

Using an example from everyday life, describe two factors that interact to have an effect on some kind of measurable quantity.

Question 51

Compare and contrast "matching" and "adjusting" with respect to experimental design and data analysis. What do they both do, and in which ways do they differ?

Question 52

When determining the required sample size for an experiment, there are two major concepts that were discussed. Describe each of these and distinguish between them. Also, describe several factors that limit sample size and explain why.

Question 53

Recall that doing many statistical tests increases the overall chance of a Type I error. If we used an a = 0.05 threshold for each test and the null hypothesis is actually true for all the tests, how many tests would we need to do before our risk of making at least one Type I error reaches 50%? Show all the steps in your calculation and remember that logarithms can be useful for solving equations with exponents.

Question 54

Recall that doing many statistical tests increases the overall chance of a Type I error. Consider a situation in which we test 1,200 genes for their influence on a measured factor. If we use an a = 0.05 threshold for each test and obtain 83 significant results, how would we interpret this, and what would the next step be? Be sure to make a quantitative statement about these results in reference to the technical phrase "false discovery rate" in your answer.

Question 55

Consider a study in which we wish to see whether the lengths of fish are related to the size of the lake in which they live. Unfortunately, the amount of dissolved nitrogen in the water from farm runoff is also thought to influence fish size. Imagine we are provided with the data shown in the table and wish to perform a matched design study where we compare the sizes of the fish in the largest four lakes to those in the smallest four lakes.
​
​
(a) Which fish should be in each of the two observation groups, and how should they be matched?
(b) After matching, what is the mean and standard error of the differences in the sizes of the fish?
(c) Based on your values in part (b), does there seem to be a difference in the mean size of the fish in the small and large lakes?

Question 56

Consider a study in which we wish to see whether the lengths of fish are related to the size of the lake in which they live. Unfortunately, the amount of dissolved nitrogen in the water from farm runoff is also thought to influence fish size. Imagine we are provided with the data shown in the table and wish to perform a matched design study where we compare the sizes of the fish in the largest four lakes to those in the smallest four lakes.
​
​
(a) Which fish should be in each of the two observation groups and how should they be matched?
(b) After matching, what is the mean and standard error of the differences in the sizes of the fish?
(c) Based on your values in part (b), does there seem to be a difference in the mean size of the fish in the small and large lakes?