Publication Date

8-2012

Type of Culminating Activity

Thesis

Degree Title

Master of Science in Instructional & Performance Technology

Department

Instructional and Performance Technology

Supervisory Committee Chair

Seung Youn Chyung, Ed.D.

Abstract

Behavior-based (Level 3) and results-based (Level 4) evaluations of training interventions can provide organizations with substantive proof of the value of those interventions. Training professionals have long acknowledged the necessity of conducting such evaluations, yet Level 3 evaluations are conducted for only about half of all training interventions and Level 4 for about one-third. This research examined the frequency with which training professionals currently conduct Level 3 and Level 4 evaluations, their perceptions on the utility of Level 3 and Level 4 evaluations, and the factors that facilitate or obstruct their attempts to perform such evaluations.

The research was conducted using Brinkerhoff's Success Case Method as its framework. Sixty-eight training professionals completed an online survey to assess their usage and understanding of Level 3 and Level 4 evaluations, indicate their success or non-success to conduct these evaluations, and rate the factors which may have contributed to their success or non-success. Twenty-two of the survey participants were interviewed to collect more in-depth information about their perceptions of these factors and how they impacted attempts to evaluate training interventions at their organizations.

The survey found that 43.47% of the training professionals surveyed conducted Level 3 evaluations at least some of the time, with only 26.08% conducting them on more than 60% of their training interventions. At Level 4, 18.41% of the training professionals conducted evaluations at least some of the time, with 13.15% conducting them on more than 60% of their training interventions. The three key factors identified by survey respondents as impacting their ability to conduct Level 3 and Level 4 evaluations were the availability of resources such as time and personnel, managerial support, and expertise in evaluative methodology. The interview data supported the survey findings but also showed that expertise also extended to an understanding of what a results-based evaluation can measure and how it is relevant to an organization; if the training professional cannot clarify the relevance of evaluating training interventions in terms of organizational goals, the organization may not see the value in expending the resources needed to conduct evaluations.

The research findings indicated a need to further explore how training professionals interpret Level 3 and Level 4 and how they can better develop their evaluative expertise, which in turn may increase the effectiveness in gaining organizational support for evaluation efforts.

Download

Share

COinS