DTIC ADA362163: Level-Three Training Evaluation: The Coast Guard's Method and Design,

8 Pages·0.43 MB·English

by  Defense Technical Information Center

Level-Three Training Evaluation: The Coast Guard's Method and Design
AB-21 - Symposium
Level-Three Training Evaluation: The ,
Coast Guard's Method and Design
Gene Adgate, Mark Rutherford, &
Christopher Hall
United States Coast Guard
Introduction
It is axiomatic that evaluation plays a major role in all phases of the systematic approach
to improving human performance. Implementing performance interventions without
following up through evaluation means that positive results, though hoped for, cannot be
directly verified. In such a case, an organization may train, create job aids, buy new
equipment or institute any number of other interventions without the means to determine
whether these "investments" actually improved performance and provided a reasonable
"return."
Because training is unquestionably among its more resource-intensive and pervasive
interventions, the Coast Guard recently analyzed the evaluation of training to establish
policies and procedures to ensure that its training efforts are effective. This paper
describes the Coast Guard's newly implemented training evaluation policy and
associated responsibilities and infrastructure.
The Case of Action
The Coast Guard sponsors over 700 resident and non-resident training courses offered by
various training providers. The term "training provider" in this paper refers to any
source that delivers training to Coast Guard members (e.g., Coast Guard Training
Centers, contracted training vendors, and other government agencies and military
services). Historically, evaluation methods have varied among training providers with
many providers conducting only the most rudimentary student critiques. Moreover, data
received from evaluations were not sufficiently communicated to Program Managers or
Training Consultants, the organization-level reviewers who make key decisions
regarding training resource allocation. Without a steady stream of evaluation data, these
reviewers have little or no information to identify trends and ensure training is "hitting
the mark' in terms of improving or enabling performance.
Seeking to remedy this situation, the Coast Guard adopted the widely accepted four-level
evaluation model first introduced by Dr. Donald Kirkpatrick some four decades ago. A
Level-Three Training Evaluation: The Coast Guard's Method and Design
brief explanation of these levels is included in Figure 1. The levels are ordered in the
sequence in which they are assessed; each level provides valuable feedback on distinct
aspects of the linkage between training and performance. The first two levels, referred to
as "Internal evaluations," are internal to the training providers and directly affect the
design, development, delivery and administration of training. The second two levels,
known as "external evaluations," are external to the training providers and focus on the
impact of training on job and organizational performance.
The Four Levels of Training Evaluation
(The Kirkpatrick Model)
Level Definition Advantages Disadvantages
Level 1- Measures trainees' Low cost and easy Only reflects a
opinions about the to administer, quick reading of the
"Reaction" course. This Is the Provides insights to participant while
most common way participant's they are still in the
to evaluate student personal feelings class. Results
reaction to the about the course. should not be used
course and provides Provides quick as a solid basis for
a measure of feedback on changing the
immediate customer successes and educational content
satisfaction with failures to the or strategy
content, delivery, training provider.
and environmental
factors. Often
referred to as "Smile
Sheets."
Level 2- Measures how well Compared to level Requires more time
participants have 1, this provides and money than
"Learning" mastered the course more compelling level 1. Also
objectives. Can evidence of whether requires greater
include tests of the training program insight to the
performance works, evaluation process
immediately before to develop valid
and after the course, measures of
learning.
2
Level-Three Training Evaluation: The Coast Guard's Method and Design
Level 3- Assesses practical Provides stronger Requires
value of training, evidence that the significantly more
"Behavior" Measures how well investment in investment of time
the knowledge, training yields the and money.
skills, and/or values desired return. If Requires in-depth
from the course are designed properly, insight into
used in the job. can also identify performance
Typically measured barriers and interventions and
3-6 months after the obstacles to root causes of
course, improved performance
performance. deficiencies.
Level 4- Measures Provides strong Substantial levels ot
performance evidence that investment and
"Results" improvement, training program hal expertise are
quality impact on required to develop
improvements, and organization. level 4. Often hard
cost savings to the Addresses whether to decide whether or
organization. the performance is not this level is
Measures the return important to the required. Linkage
on investment of the organization's from training to
training course. bottom line (e.g., org'l results is hard
production, safety, to establish.
sales).
Figure 1: The Four Levels of Training Evaluation
Policy Analysis
Substantial research and analysis were conducted prior to developing any polices or
making any sweeping changes to the existing informal training evaluation performance
consultants and a full-time graduate student spent over a year examining related
literature and best practices. Below are observations and conclusions based on the
research, benchmarking, and analysis that helped shape a new Coast Guard-specific
training evaluation policy and infrastructure.
1. Level- I and -2 evaluations are clearly the responsibility of the training providers,
who are most knowledgeable of the training environment and the trainees' reactions.
The training providers have sole control over how they administer and deliver the
course objectives and whether or not the participants acquire the knowledge, skills, or
values during training. Coast Guard Training Centers are staffed with civilian and
military professionals well versed in the Instructional Systems Design (ISD) process and
are very capable of handling levels l and 2. General guidelines are all that is needed to
ensure uniformity of practice.
2. Level-3 evaluations require a standardized process to ensure the results are
communicated consistently and regularly to the appropriate decision-makers. Under the
3
Level-Three Training Evaluation: The Coast Guard's Method and Design
existing system, many training providers were not even conducting level-3 evaluations.
Where these evaluations were conducted, there were problems including:
0 There were no criteria for using and analyzing evaluation data.
0 Questions and evaluation forms were non-standard and often did not accurately
reflect the course content.
0 The evaluations were not built into the training process and therefore were
inconsistent. Identifying negative trends and making decisions on remedial performance
interventions were virtually impossible.
0 Program Managers and Training Consultants-those who make decisions affecting
whether courses are maintained, deleted, or modified-did not receive evaluation data.
3. Some level-4 evaluations are in place. There is a significant body of literature that
indicates level 4 is not required for all types of training. Moreover, it is difficult to
isolate the contribution of any one determinant such as training in achieving
organizational results. Finally, measurement of some training courses and their
competency objectives would either be impossible or would yield invalid and
inconclusive results. For example, even if Coast Guard EMTs never used CPR, the
course would still be deemed necessary. Level 4 requires more research before making
changes. As with levels I and 2, general guidelines will assist Program Managers in
continuing to assess the ultimate worth of training.
A Focus on Level 3
In view of the research results, it was clear that special emphasis needed to be placed on
level-3 evaluation efforts, which were not meeting the needs of the Coast Guard's
training system. Correcting this shortfall required a completely different approach. A
formal mechanism was established that revolved around a standard survey instrument
that would assess the on-the-job impact of training. The entire process to collect course
data, distribute post-graduation surveys, collect responses, and report results to
appropriate consumers would be centralized to provide for standardization, efficiency,
and consistency.
Level-3 Evaluation Administration
Due to its resident expertise in surveys and capability for mass mail distribution, the
Coast Guard Institute, located in Oklahoma City, was selected as the administrative hub
for level-3 evaluations. To this end, the Coast Guard Institute is now responsible for
maintaining a database of course completion dates and graduates to trigger mailing of
the right surveys to the right people at the right times. In this database, the Institute will
maintain a library of instruments that have been tailored to individual courses. Six
months after the completion of a course, the Institute will send a survey to each trainee
and the trainee's supervisor. Concerns about sampling validity are minimal since the
4
Level-Three Training Evaluation: The Coast Guard's Method and Design
entire target population will be surveyed.
Once the survey is returned to the institute, the data will be captured in a report to
Program Managers and Training Consultants providing frequency of responses.
Program Managers and Training Consultants can request additional reports to correlate
specific variables (e.g., unit to unit, supervisors to trainees, class to class comparisons).
These reports will be used to make decisions on whether in-depth analysis is required to
determine causes for inadequate transfer of training to the job. Undesirable survey
results would require such an analysis since a variety of barriers (possibly having
nothing to do with the way the training was conducted) might prevent training from
having its intended impact.
To support this policy administration, two new data-entry positions will be added to the
Institute's staff Moreover, an evaluation expert will be added to the Coast Guard
Headquarters Office of Training and Performance Consulting to oversee the overall
process
The Level-3 Survey Instrument
The team of survey developers recognized the unfeasibility of using the instrument to
identify the myriad possible barriers to human performance. Instead, the level-3
instrument will serve as an indicator or "dashboard gauge" to help identify areas
requiring a more detailed analysis of the transfer of training to job behavior.
Initially, it was thought that the course objectives themselves could be directly inserted
into the instrument; however, this resulted in redundancies and poorly worded items
unsuitable for a survey. As a result, training providers were charged with development
of "performance indicators" that reflect the ultimate job performance objectives of each
course. The survey framework into which these performance indicators fit is depicted in
Figure2 . Designated respondents receive a cover sheet with the survey explaining the
instrument and providing instructions for completion.
Conclusion
The successful implementation of the four-level evaluation system required a new policy
facilitated by an innovative infrastructure. The policy outlines responsibilities at all
levels in the evaluation process providing informal guidelines for levels 1, 2 and 4 and
formal mandates for level 3. The Coast Guard Institute provides primary infrastructure
support. The new evaluation policy will make great strides in validating the
effectiveness of the Coast Guard's training system. There is no doubt that the
combination of a strong internal evaluation system with an efficient and well-managed
external evaluation system will ensure that Coast Guard training effectively provides
essential knowledge and skills needed to meet the Service's challenging, multi-mission
demands.
Post-Training Survey
5
Level-Three Training Evaluation: The Coast Guard's Method and Design
Section 1: Task List
Do you perform the following? YES NO
If yes, rate your confidence: HIGH -------------- W
Performance Indicator A 1 2 3 4 5 6
Performance Indicator B... etc. F 7 3 4 5 6
Section 2: Training Benefit
1 - Training provided complete knowledge/skills to proficiently perform the
task.
2 - Training provided a strong base to proficiently perform the task.
3 - Training was of little or no benefit to proficiently perform the task.
4 - Task performed at this unit, but my duties do not require me to complete
this task.
5 - Task not performed at this unit.
Which of the statements above is most true of the following tasks?
Performance Indicator A 1 2 3 4 5 6
Performance Indicator B... etc. T 2 T T T 6
Section 3: Comments
1. Are there any tasks performed at your unit that you feel should have been
addressed in this course?
2. Is there anything that prevents you from becoming proficient in the tasks associated
with this course (e.g., tasks are automated, do not perform the tasks at your level of rank,
outdated procedures or policies taught in the course)?
Figure 2: Sample of Level-3 Survey Instrument
6
Level-Three Training Evaluation: The Coast Guard's Method and Design
7
INTERNET DOCUMENT INFORMATION FORM
A. Report Title: Level-Three Training Evaluation: The Coast
Guard's Method and Design
B. DATE Report Downloaded From the Internet 4/21/99
C. Report's Point of Contact: (Name, Organization, Address,
Office Symbol, & Ph #): Navy Education and Training
Professional Development and
Technology Center
Navy Advancement Center Dept
Dr. Grover Diesel, [phone]
6490 Saufley Field Road
Pensacola, FL 32509-5237
D. Currently Applicable Classification Level: Unclassified
E. Distribution Statement A: Approved for Public Release
F. The foregoing information was compiled and provided by:
DTIC-OCA, Initials: VM_ Preparation Date: 4/22/99
The foregoing information should exactly correspond to the Title, Report Number, and the Date on
the accompanying report document. If there are mismatches, or other questions, contact the
above OCA Representative for resolution.