Based on the material provided in the reading, interpret the role of the program in assessing change.

Question

Quizplus · Accepted Answer

The Answer is The first step in understanding the role the program might play in effecting change is to be clear about what change is intended, desired, and/or likely, given the program's logic model or the theory of change for the program. It helps if the theory of change is sufficiently detailed to specify the program elements or actions that are used to effect types of changes. Even when there is a clear and specific theory of change or logic model, before developing an evaluation methodology it is important to clarify the specific desired outcomes expected by the program. It is also important to clarify what the most important potential outcomes of the program might be, because we rarely have enough resources to conduct a fully comprehensive evaluation that answers all the questions that stakeholders might have.
Having a good partnership with direct and open communication is particularly helpful at this point, to consider what program "effects" might occur under different sets of circumstances--for different participants, for those not directly participating in the program (e.g., parents might be affected by a program that is explicitly designed to improve the behavior of children), for those who are culturally different from the mainstream, or for participants of different genders or ages. Such discussions should lead to the identification or development of the specific indicators or measures to allow us to determine to what extent those changes are occurring. Measures need to be reliable and valid as much as possible, with the recognition that there are times when a desired outcome is not easily measured and not clearly defined. An assessment of change generally implies that we measure the construct in question at least 2 times, so we can see if there is a difference between a "pretest" or baseline value and a "posttest" value. There are many variations on a "pretest-posttest" design that involve such measurement at two points in time and allow a direct assessment of change among the program participants. However, there are many cases when having a "pretest" does not make sense or is not possible. When assessing the difference between observations at different times (i.e., pre- vs. posttest) or among different groups (e.g., treatment vs. comparison group), it is desirable to determine if that difference is statistically significant, or not likely to occur simply by chance. The use of the term "desirable" is very deliberate, although some researchers would suggest that it is "essential," because without statistical significance, you should not consider the difference as a real difference. Since we assume that readers of this book have a basic background in research design and statistical analysis, we will not spend time explaining the meaning of statistical significance. However, it is important to recognize that there are many times when it is unlikely that a difference will be statistically significant, because of a low number of participants, unreliable measures or interventions (different participants get different "treatments"), highly variable samples of participants or a host of other reasons. In those cases, there may be a "real difference," not due to chance, but we just cannot detect it. Yet, if we have a sizable apparent (but not statistically significant) difference, under some circumstances this might be appropriately used by a program manager as a reason to modify the program.

Based on the Material Provided in the Reading, Interpret the Role