Deck 10: Training Evaluation

Question 1

Affective reaction measures the level of perceived usefulness of a training program.

Answer 1

False

Question 2

Formative evaluations provide data about the worthiness or effectiveness of a training program.

Answer 2

False

Question 3

Those organizations that perform training evaluations tend to focus their metrics on evaluating training benefits associated with return on investment (ROI), with far fewer focusing on behaviour and reaction metrics.

Answer 3

False

Question 4

Measures of self-efficacy tend to focus on a trainee's perceived level of support in transferring newly learned skills to the job.

Question 5

One of the more rigorous evaluation techniques involves measuring the level of satisfaction by trainees after completing a training program.

Question 6

For both self-report and observational data, it is important that the measures focus on specific behaviours as opposed to general behaviours because specific behaviours are more likely to be valid and accurate.

Question 7

Causal evaluations require more sophisticated experimental and statistical procedures versus the less sophisticated descriptive evaluations.

Question 8

Studies of training professionals show the more sophisticated training evaluations are rarely used because they are perceived to be too complicated to implement, take too long to design, and simply cost too much money.

Question 9

Declarative learning refers to the acquisition of facts and information and is by far the most frequently assessed learning measure.

Question 10

Training evaluation is a process that helps organizations determine if expected training outcomes actually materialize.

Question 11

One of the primary advantages of the Decision-Based Evaluation model over the Kirkpatrick model is that it allows different variables to be measured depending on the target of the evaluation process.

Question 12

The COMA model and the Decision-Based Evaluation model are the most widely used training evaluation methods.

Question 13

The main reason procedural learning measures are NOT often used is centred on the level of complexity required to properly assess procedural learning.

Question 14

According to the Conference Board of Canada, only 46 percent of Canadian organizations undertook some type of training evaluation in 2010. However, in 2016-17 there was evidence to suggest Canadian companies were improving on the 2010 evaluation results.

Question 15

A question such as "Has the training program resulted in payoffs for the organization?" is typically associated with summative evaluations.

Question 16

Kirkpatrick's hierarchical model has four levels. Of these four levels, a survey provided to trainees after training is often associated with collecting data for calculating behavioural changes at work.

Question 17

Training evaluation is undertaken for a variety of reasons. Some of the most salient include identifying the costs and benefits of a particular training initiative and determining if expected training outcomes materialized.

Question 18

When measuring trainees' "reactions" to a training program, trainers often use questionnaires, focus groups, and interviews.

Question 19

One of the main implications of the Kirkpatrick model of evaluation is the interdependencies among the four levels of evaluation. In practical terms, "(L1) should cause trainee learning (L2), which in turn should cause behaviour (L3), and so on."

Question 20

The "pre-post design with control group" data collection method answers two basic questions: Did trainees change? and Was training responsible for that change?

Question 21

Which of the following is NOT a criticism of Kirkpatrick's hierarchical evaluation model?

A) several authors have challenged the validity of the hierarchical nature of the model
B) the model does not offer a solution if and when a program fails to deliver on objectives
C) the model requires that all training evaluation processes use the same variables
D) the model offers a fairly straightforward framework for evaluating training

Question 22

Which of the following statements is NOT congruent with conclusions in theChapter-opening vignette on Bell Canada?

A) classroom training seemed to impact learning outcomes the most
B) the traditional training method was ineffective
C) changing the instructors had no effect on learning outcomes
D) the use of a well-developed instruction booklet had greater effect

Question 23

Utility reaction measures demonstrate some relationship to higher-level outcomes.

Question 24

The measurement of a training program's ability to meet managerial expectations is called soft data.

Question 25

Which of the following is a key advantage of the Decision-Based Evaluation (DBE) model over the other two evaluation models described in the textbook?

A) The DBE model has significant empirical evidence supporting it.
B) Unlike the other models, the DBE model provides the specific variables to be used.
C) DBE allows the program evaluator to modify the variables to be used in the evaluation process to meet specific situations.
D) DBE is much less complex than both COMA and Kirkpatrick's model.

Question 26

You are putting together a short report addressing why many training programs are poorly evaluated by organizations beyond Kirkpatrick's "level one." Which of the following is NOT likely to be in your report?

A) most training managers have been trained on how to properly evaluate programs and senior management often request it
B) management does not request training evaluation and does not provide funding for it
C) trainers and human resource departments (HR) may not be interested in the results, as outcomes may reflect poorly on the HR department
D) lack of time for training evaluation

Question 27

Which of the following statements best describes the state of training evaluation in Canada according to the Conference Board of Canada?

A) More than 90 percent of Canadian companies assessed training in some fashion in 2016-17.
B) The level of sophistication of training evaluation methods has decreased over the last few years.
C) Starting in 2008 and thereafter the proportion of organizations conducting evaluations dropped to half or less.
D) The most common methods of training evaluations are results and behaviour focused.

Question 28

Carlos, an instructor at a local university, wants to measure the effectiveness of his training sessions in terms of payoffs for his employer. Which of the following evaluation methods would you recommend to Carlos?

A) transfer of training climate evaluation method
B) Kirkpatrick's evaluation scheme, level 1
C) summative evaluation method
D) formative evaluation method

Question 29

For the variable of organizational results, which of the following questions is typical of one used in a training evaluation survey?

A) How much has the organization's turnover rate decreased as a result of training?
B) How many times has the manager recognized employees for extra effort displayed?
C) How many of your training managers can identify Kirkpatrick's four levels of evaluation?
D) How satisfied were you with the content of the training program and how much will you apply on the job?

Question 30

Which training evaluation model is dominant in both research and practical experience?

A) Kraiger's hierarchical model
B) Kirkland's hierarchical model
C) Gagné's hierarchical model
D) Kirkpatrick's hierarchical model

Question 31

What are the two broad categories of barriers to training evaluation?

A) formative and summative
B) pragmatic and political
C) Kirkpatrick and COMA
D) reaction and behavioural-based

Question 32

With the Internal Referencing Strategy design, data are collected exclusively from the trained group using a post-only design.

Question 33

The COMA evaluation model suggests measurement of variables that fall into four categories rather than relying only on declarative and reaction learning outcomes. Which of the following is NOT one of those four categories?

A) cognitive variables
B) organizational payoff variables
C) motivation variables
D) attitudes variables

Question 34

Which two types of "reaction measures" have received the most attention in the literature?

A) descriptive and causal
B) perceived and anticipated
C) declarative and procedural
D) affective and utility

Question 35

Company ABC noticed an increase in customer satisfaction after employees completed a new customer satisfaction training program. As the VP of human resources, you have been asked to determine if in fact the training program led to the increase in customer satisfaction results. Which of the following training evaluation approaches would you take?

A) formative evaluation
B) summative evaluation
C) descriptive evaluation
D) causal evaluation

Question 36

Prior to 2008, the majority (three-quarters or more) of organizations conducted evaluations. Starting in 2008 and thereafter that proportion increased to 90 percent.

Question 37

The COMA model is intended principally for conducting summative evaluations.

Question 38

Your company has just spent $100 000 on a new training program for new employees. You have been asked to calculate the return on investment on this new training program. Which level of Kirkpatrick's general model would you use to inform your answer?

A) Level 1
B) Level 5
C) Level 4
D) Level 3

Question 39

According to the textbook, which of the following statements is NOT correct when looking at the meaning and purpose of training evaluation processes?

A) diagnose the strengths and weakness of a program and pinpoint needed improvements
B) use evaluation information to justify and reinforce the value and credibility of the training function to the organization
C) determine the cost benefits of a program and help ascertain which program or training technique is most cost-effective
D) determine if the training program has given trainees full top management support via objective metrics

Question 40

Which of the following is considered to be a political barrier to training evaluation?

A) senior management may not be stressing the need for training programs to be evaluated against goals and objectives
B) to take resources away to conduct evaluations may be seen as counterproductive
C) proper assessment of a training program requires the use of two precious commodities: time and money
D) taking on the training evaluation task requires knowledge of evaluation models, research design, measurement, questionnaire construction, and data analysis

Question 41

J. Bruce Tracey, Scott Tannenbaum, and Michael Kavanagh proposed a scale to measure which of the following organizational issues?

A) rewards and incentives linkage to behaviour changes
B) the culture of the organization as it relates to continuous learning
C) the perceived level of management support and peer support
D) the level of overall organizational support

Question 42

What outcomes are measured using the Internal Referencing Strategy model?

A) reactions and learning
B) relevant and nonrelevant
C) accurate and inaccurate
D) correct and incorrect

Question 43

You are finding it extremely Difficult to link organizational bottom-line improvements to a specific training program. What is an acceptable process that you may use to justify the training program given the lack of linkage between training and organizational outcomes?

A) return on investment
B) return on expectations
C) return on training sunk costs
D) return on training expenses

Question 44

Why is a "post-only" design the most widely used data collection method?

A) it is the easiest data collection method
B) it establishes causality
C) it is statistically the most comprehensive data collection method
D) it is most reliable in terms of conclusions

Question 45

Transfer motivation is critical in supporting transfer of training content. One technique uses the expectancy theory measuring three variables: valence, instrumentality, and expectancies. Which of the following statements is associated with the "instrumentality" variable?

A) the positive or negative consequences of transfer
B) the attractiveness of transfer outcomes
C) the probability that transfer will result in successful performance
D) the level of management support

Question 46

You need to evaluate the effect of a training program on the organization and it will be Difficult or impossible to adequately assess the impact of training directly either with hard or soft data. What will be a good alternative?

A) affective reactions
B) utility reactions
C) return on expectations
D) organizational perceptions

Question 47

Which statement reflects the concept of perceived support?

A) "I believe my organization will support me when I apply newly learned skills."
B) "I am somewhat confident my manager will support me when I apply newly learned skills."
C) "I believe my manager most likely will support me when I apply newly learned skills."
D) "My manager coaches me on how best to apply newly learned skills."

Question 48

What kind of research design would you use if you had very limited resources and were concerned only with measuring improvements in a group of trained employees?

A) quantitative design
B) experimental design
C) quasi-experimental design
D) non-experimental design

Question 49

Which theory is an important technique used to assess motivation to transfer?

A) conditioning theory
B) social cognitive theory
C) ERG theory
D) expectancy theory

Question 50

Which of the following is NOT a basic approach for measuring post-training behaviour?

A) observations
B) production data
C) focus groups
D) self-reports

Question 51

What is the most significant advantage of conducting learning tests with trainees?

A) When budgets are being reduced, learning tests give objective evidence that they enhance performance and so training should continue.
B) If trainees consistently score low on some aspect of the course, they will need to do more work.
C) If insufficient time has been devoted to developing learning tests, the results could be arbitrarily inflated.
D) In cases of accidents or litigation, the employer can prove the employee was trained adequately.

Question 52

Immediate post-training questionnaire measures remain very popular with trainers. Which of the following is NOT a reason for this popularity?

A) they are easy to collect and analyze, and are easily understood by managers and employees
B) they provide trainers with immediate feedback
C) they are an easy and convenient mechanism for measuring levels 1, 2, and 3 of the Kirkpatrick evaluation model
D) they are an easy and convenient mechanism for measuring affective trainee reactions

Question 53

Which type of learning measure is associated with positive training transfer?

A) declarative learning
B) procedural learning
C) organizational learning
D) observational learning

Question 54

What training evaluation design is used in the Internal Referencing Strategy model?

A) pre-only design
B) pre-post design
C) time series design
D) post-only design

Question 55

Which of the following evaluation and data collection methods would offer the strongest conclusions on the effectiveness of a training program?

A) single-group design
B) pre-post design
C) time series design with control group
D) post-only design

Question 56

A training manager is concerned with whether the results of training are based on application of learned skills or some other external/internal variable. Which type of data collection design would address these concerns?

A) pre-post
B) post only
C) time series
D) time series with control group

Question 57

Your company wants to use the observation method to measure post-training behaviour changes. What advice would you provide your company about using this method?

A) instruct management to limit contact with trainees
B) instruct management to focus on specific behaviours displayed by trainees
C) allow management to use judgment in rating performance
D) instruct management to focus on general behaviours displayed by the trainee

Question 58

What factors need to be determined according to the decision-based evaluation model?

A) target, focus, design
B) focus, source, design
C) target, variables, methods
D) target, focus, methods

Question 59

Which common shortcoming is shared by all three descriptive training evaluation models described in the textbook?

A) the results and data collected are simply not applicable to most training programs
B) the collection of the data is almost always very expensive and time consuming
C) the results and data collection methods do not account for external variables that may contaminate results
D) implementation of descriptive evaluation models requires mastery of statistical analysis techniques

Question 60

A new and very expensive training program will be implemented in your organization and it is very important to find out if the training has a positive effect on trainees' job performance. You are responsible for designing the evaluation. What training evaluation data collection design will you recommend?

A) post-test only
B) pre-post design
C) pre-post with control
D) time series

Question 61

Compare and contrast the concepts of perceived support and anticipated support.

Question 62

Describe the two types of barriers to training evaluation and provide examples of each type of barrier.

Question 63

a) Identify the four levels of Kirkpatrick's hierarchical model of evaluation and briefly explain the concerns with it.?b) What is the fundamental difference between Level 4 and Level 5 (the new addition to the original four levels of Kirkpatrick's model)?

Question 64

Identify and briefly describe the four categories of the COMA training evaluation model.

Question 65

What is the Internal Referencing Strategy (IRS) and how does it work? Be clear about how it is used to evaluate a training program.

Question 66

Explain the terms declarative learning and procedural learning, describe how each one can be assessed, and state which one is most important to training transfer.

Question 67

Why would management be interested in a revision of Kirkpatrick's model that includes a fifth element, and are there situations when calculating a return on a training investment is not practical?

Question 68

Briefly explain the differences between non-experimental designs, experimental designs, and quasi-experimental evaluation designs.