ERIC EJ859305: The Effect of Feedback on the Accuracy of Checklist Completion during Instrument Flight Training

2009·0.92 MB·English

by  ERIC

JOURNALOFAPPLIEDBEHAVIORANALYSIS 2009, 42, 497–509 NUMBER3 (FALL2009)
THE EFFECT OF FEEDBACK ON THE ACCURACY OF CHECKLIST
COMPLETION DURING INSTRUMENT FLIGHT TRAINING
WILLIAM G. RANTZ, ALYCE M. DICKINSON, GILBERT A. SINCLAIR, AND
RON VAN HOUTEN
WESTERNMICHIGANUNIVERSITY
This study examined whether pilots completed airplane checklists more accurately when they
receivepostflightgraphicandverbalfeedback.Participantswere8collegestudentswhoarepilots
with an instrument rating. The task consisted of flying a designated flight pattern using a
personalcomputeraviationtrainingdevice(PCATD).Thedependentvariableswerethenumber
of checklist items completed correctly. A multiple baseline design across pairs of participants
with withdrawal of treatment was employed in this study. During baseline, participants were
given postflight technical feedback. During intervention, participants were given postflight
graphic feedback on checklist use and praise for improvements along with technical feedback.
Theinterventionproducednearperfectchecklistperformance,whichwasmaintainedfollowing
areturntothe baseline conditions.
DESCRIPTORS: feedback,flight checklists, pilot training
_______________________________________________________________________________
In aviation, the checklist is used during with other cockpits or with newer generation
differentsegmentsofflighttosequencespecific, cockpits. In addition, the checklists can be very
critical tasks and aircraft configuration adjust- long. For example, on some checklists, the
mentsthatcorrespondtospecificenvironmental ‘‘before engine start’’ subsection has 76 items
demands (Degani & Wiener, 1990). It is for the first flight of the day and 37 items for
divided into subsections with task checklists subsequent flight segments (Degani & Wiener,
that correspond to all flight segments and, in 1990). Thus, it is not surprising that many
particular, critical segments such as takeoff, aviation experts have addressed their impor-
approach, and landing. tance and design, as well as the practices and
The complexity of these task checklists policies that surround their use (Adamski &
cannot be overstated. Standard procedures Stahl, 1997; Degani, 1992, 2002; Degani &
common to some cockpits are not compatible Wiener; Gross, 1995; Turner, 2001; U.S.
Federal Aviation Administration [FAA], 1995,
2000). Even so, the incorrect use of flight
Reprints andmaterials including thechecklists, techni-
cal flight pattern parameters and narration, checklist checklists is still often cited as the probable
performance protocol, and checklist observation form cause oracontributingfactortoalargenumber
can be obtained from William Rantz, Western Michigan
of crashes (Degani; Degani & Wiener, 1993;
University College of Aviation, 237 Helmer Road, Battle
Creek, Michigan 49015 (e-mail: William.rantz@wmich. Diez, Boehm-Davis, & Holt, 2003; Turner).
edu). Similarly, many investigations by the U.S.
We thank colleagues and students who provided
National Transportation Safety Board (NTSB)
assistance and support that made this study possible.
have revealedthattheaircraft werenotproperly
These include Brad Huitema, Vladimir Risukhin, John
Austin, Sean Laraway, Susan Syznerski, Axel Anderson, configuredforflight,whichusuallyresultsfrom
Brandon Jones, and Alex Merk. We also thank Dave improper use of checklists (NTSB, 1969, 1975,
Gaurav and James Burgess for their technical support in
1982, 1988a, 1988b, 1989, 1990, 1997).
theexperimental laboratory.
Address correspondence to Ron Van Houten, Depart- Studies by Lautmann and Gallimore (1987)
ment of Psychology, Western Michigan University, and Helmreich, Wilhelm, Klinect, and Merritt
Kalamazoo, Michigan 49008 (e-mail: ron.vanhouten@
(2001) provide more direct evidence of im-
wmich.edu).
doi:10.1901/jaba.2009.42-497 proper use of checklists by flight crews. In a
497
498 WILLIAM G. RANTZ et al.
study funded by the Boeing aircraft manufac- thatthepromotionofapositiveattitudetoward
turer, Lautmann and Gallimore surveyed 12 the use of the checklist procedure was an
airlines and concluded that procedural errors important element that was often overlooked.
involving use of the checklist contributed to a Regardless, an extensive search of the aviation
substantial number of aircraft crashes and checklist literature did not reveal any studies
incidents. that have examined whether behavioral inter-
In an effort to identify particular errors flight ventions could increase the appropriate use of
crewscommit,theU.S.NationalAeronauticsand flight checklists.
Space Administration sponsored a series of Although a number of behavioral studies
studies in which crews were observed while have employed checklists as part of or the sole
flying. Observers using the line-oriented safety independent variable in a treatment plan
audit (LOSA) that recorded checklist behaviors (Anderson, Crowell, Hantula, & Siroky, 1988;
throughout the flight (Helmreich, Klinect, Wil- Austin, Weatherly, & Gravina, 2005; Bacon,
helm, & Jones, 1999; Helmreich et al., 2001) Fulton, & Malott, 1982; Crowell, Anderson,
recorded crew errors. Between 1997 and 1998, Abel, & Sergio, 1988; Shier, Rae, & Austin,
LOSAswereconductedatthreeairlineswith184 2003) only a few studies have focused on
flightcrewson314flightsegments(Helmreichet checklist use as a dependent variable (Burgio,
al.). Seventy-three percent of the flight crews Whitman, & Reid, 1983).
committed errors. The number of errors ranged Inaviation,incorrectchecklistusecanleadto
from 0 to 14 per flight, with a mean of 2. Rule- fatal consequences. In addition, the completion
compliance errors were the most frequently of checklists during flight is more behaviorally
occurringerrors,accountingfor54%ofallerrors challenging than in the settings in which
(Helmreich et al.). Checklist errors constituted checklist use has been evaluated, due to
the highest number of errors in this category. constantly changing environmental demands,
Despitewidespreadrecognitionthatchecklist distractions, and schedule pressures. For exam-
errors occurred relatively frequently and were ple, in one fatal crash, thetaxi checklist was not
major contributing factors to many crashes, the completedbecauseofseveralinterruptions(new
design of checklists ‘‘escaped the scrutiny of the weather information, checking aircraft and
human factors profession’’ until the 1990s runway data; Degani & Wiener, 1990). Yet,
(Degani & Wiener, 1993, p. 28). Degani and to date, no study has examined whether
Wiener (1990, 1993) observed flight crews behavioral interventions can improve checklist
while flying, interviewed flights crews from use. Performance in aircraft simulators is one
sevenmajorU.S.airlines,andanalyzedhowthe method that could be used to evaluate pilots’
design of checklists contributed to aircraft use of checklists. In recent decades, personal
crashes and incidents that were reported in computer aviation training devices (PCATDs)
three aviation databases. Their analytic guide- have emerged as an effective, low-cost platform
lines became the industry standard (Patterson, for training instrument flight skills. For the last
Render, & Ebright, 2002). 10years,researchershavedemonstratedpositive
AlthoughDeganiandWiener(1990)didnot transfer from PCATDs to the actual aircraft
pursue the behavioral factors that influence (Taylor et al., 1999). Thus, up to 10 hr of
checklist use, they recognized their importance, simulated flight experience gained while using
indicating that safety culture issues related to an FAA-approved PCATD can be applied
supportofmisuseornonuseofchecklistswerea towardqualifyingforcertainpilotratingsunder
core problem that led some pilots to misuse the Part 61 or Part 141 of the federal aviation
checklist or not use it at all. They also noted regulations (FAA, 1997).
FEEDBACK AND CHECKLIST COMPLETION 499
One advantage of simulation training is that flight instruments to maintain straight and level
it allows more complete monitoring and flight, turn, climb, and descend while vision is
feedback of pilot behavior. For that reason, obscured by clouds, precipitation, or other
training was performed on a PCATD in this environmentalconditions.TheFAArequiresthat
experiment. Also, in the future, it is likely that pilotshaveaminimumof125flighthoursbefore
early flight training will begin in a simulator, they can obtain instrument rating; thus, all
allowing good checklist performance to be participants had these minimum flight hours.
established before further training in the The minimum 2 hr of PCATD experience
aircraft. Finally, it is likely that the operation ensured that participants had some understand-
of aircraft will increasingly be done by pilots ing of how the flight software program func-
whositataconsoleontheground.Manypilots tioned and what responses were required to
in the armed forces already fly attack and perform technical flight skills on the PCATD,
surveillance unmanned aerial vehicles (UAV) in enabling them to perform fluently sooner than
this manner, and some civilian use may follow those who would not have had such exposure.
(Bone&Bolkcom,2003).Checklistprocedures
need to be followed in these situations to Setting
prevent crashes. Operators of UAVs are trained The experimental setting was a room (3.6 m
on simulators and actual flight is indistinguish- by 4.9 m) that was used as the PCATD flight
able from simulation. Therefore, in this case, laboratory. The laboratory was located in a
transferwouldnotbeanissue.Giventhedearth hanger adjacent to Western Michigan’s Flight
of studies on behavioral interventions to Education Center in Battle Creek. Within the
improve pilots’ use of checklists, the current room, dividers restricted the vision of the
experiment examined whether postflight graph- participant to the PCATD flight simulation
ic feedback and verbal praise could increase the testing equipment.
accuracy and quality of checklist use by pilots
Apparatus
during training on a flight simulator.
PCATD equipment. The PCATD equipment
consisted of a Pentium II 300-megahertz
METHOD
processor, four megabytes of SRAM video
Participants memory, and 64 megabytes of SDRAM
Participants were 8 undergraduate students memory. Other PC equipment included a
(7 male and 1 female) who were enrolled in monitor(25.4 cmby35.6 cm)andtwoJUSTer
commercial flight courses in the aviation flight SP-660 3D speakers. Operating software was
science program at Western Michigan Univer- Microsoft Windows 95, and the simulation
sity.Theageoftheparticipantsrangedfrom20 software wasOn-Top Version8. Flight support
to 26 years. Recruitment flyers and in-class equipment for the PCATD included a Cirrus
announcements were used to notify potential yoke, a throttle quadrant, an avionics panel,
participants of the opportunity to volunteer for rudder pedals, and a number of additional
the study. Criteria for inclusion included a controlsusedto configuretheaircraft.TheOn-
privatepilotcertificate,instrumentrating,anda Top software permitted the simulation of
minimum of 2 hr PCATD experience flying several different aircraft. The aircraft that was
some type of instrument landing approach. simulated in the current study was the Cessna
Aprivatepilotcertificateandinstrumentrating C-172. The Cessna was chosen due to its vast
were prerequisites for performing the simulated popularity in the flight-training field as well as
instrumentflightpatternsusedastheexperimen- the fact that it was the primary aircraft used in
tal task. Instrument flight refers to the use of the Western Michigan training fleet.
500 WILLIAM G. RANTZ et al.
Technical flight parameters, which depicted the appropriate time of the flight segment. For
how well participants flew the designated flight example, if the checklist item required turning
patterns vertically and horizontally, were re- the heading indicator to the direction corre-
corded for each flight. The simulation software sponding to the compass reading, and the
automatically recorded these technical parame- participant turned the heading indicator (the
ters and enabled them to be printed. correct equipment) to the corresponding com-
Flight patterns. There were six different flight passheadingposition(thecorrectresponse),the
patterns.Eachflightpatternwasdividedintosix item was scored correct. However, if the
segments: (a) pretakeoff, (b) climb, (c) cruise, participant turned the heading indicator (the
(d) prelanding, (e) landing, and (f) after correct equipment) to the wrong heading (an
landing. Each pattern took approximately error), the item was scored as an error. If the
15 min to complete. To simulate actual flight participant turned the heading indicator (the
patterns realistically and to ensure that the correct equipment) to the corresponding com-
patterns were flown in a consistent way across pass heading position (the correct response) at
trials and participants, the experimenter pro- the incorrect time in the flight or checklist
vided typical air traffic control instructions sequence, the item was also scored as an error.
throughout each flight pattern. These instruc- An error was also scored if a checklist item was
tions were transmitted using an intercom. omitted.
The flight checklist. The flight checklist The first and third authors, both experienced
contained 40 checklist items divided into pilots,servedasobservers.Theyscoredchecklist
sections that corresponded to each of the six behaviors using an observation form that
flight segments. This checklist, based on the included each item and columns to check
checklist for the Cessna 172 R, was similar to whether the item was completed correctly,
the one used in Western Michigan’s flight- completed incorrectly, or omitted. The observ-
trainingcurriculum.Thechecklistwasmounted ers occupied a room adjacent to the partici-
in plain sight 25 cm to the left of the flight pant’s room and observed participants remotely
computer monitor. A paper checklist (rather viaWebcameras.The twocamerashadbuilt-in
than an electronic checklist) was used because microphones that allowed the observers to see
paper checklists are the most common type of and hear both the nonverbal and verbal
checklistusedinaviationandinotherindustries responses that were required to complete the
for complex processes (Boorman, 2001). checklist. One camera was mounted on the
computer monitor approximately 51 cm in
Dependent Variables front of the participant to capture hand and
The main dependent variable was the arm movements. The other was positioned
number of checklist items completed correctly 89 cm behind the participant to observe the
per flight. Two secondary dependent variables participant’s interaction with the flight panel.
were the percentage of total errors for each of To observe the frequencies entered into the
thesixflightsegmentsduringeachexperimental communication and navigation radios, which
phase (baseline, feedback, and withdrawal of could not be seen clearly via the cameras,
treatment)perparticipantandthepercentageof observers viewed a dual computer monitor that
baseline trials in which participants performed mimicked the participant’s computer screen.
each of the checklist items incorrectly. The settings on the computer screen mimicked
For an item to be scored correct, participants the setting on the radio stack, which was
had (a) to respond to the correct flight manipulated by the pilot. All flights were
equipment, (b) to respond appropriately with recorded and stored digitally for the purposes
respecttothatequipment,and(c)torespondat of conducting interobserver agreement checks.
FEEDBACK AND CHECKLIST COMPLETION 501
Independent Variable scheduled during this phase as well. After
Theindependentvariablewasthepresenceor instructing the participant, the experimenter
absenceofpostflight(a)graphicfeedbackonthe left the room.
total number of checklist items completed After the participant completed a flight, the
correctly per flight, (b) graphic feedback on experimenterprintedoutatechnicaldiagramof
the number of items completed correctly and the flight pattern flown by the participant. This
errors for each of the six flight segments per diagram was automatically created by the
flight, and (c) praise for improvement in the simulator software and displayed the lateral
number of checklist items completed correctly. and verticalflightpaths. Theexperimenterthen
Procedural details are described below. entered the room, gave the diagram to the
participant, and discussed the technical merits
Experimental Design of the flight, praising good performance. No
A multiple baseline design across pairs of feedback was given to the participant about the
participants with a withdrawal of treatment was use of the flight checklist. This protocol was
used. Sessions lasted approximately 1 hr, and repeated for each flight during the baseline
participants flew three different flight patterns phase.
per session. Each flight was considered a trial. Postflight checklist feedback. In addition to
The order of exposure to the six flight patterns giving participants postflight technical feed-
was randomized in blocks of six for each back, the experimenter provided feedback on
participant. A withdrawal phase was included the use of the flight checklist. After each flight,
to assess whether checklist performance would the experimenter printed a line graph that
be maintained after postflight feedback was displayed the number of correctly completed
withdrawn. items for that flight and each preceding flight,
including baseline. The experimenter also
Procedure printed a bar graph that displayed the number
Baseline. Participants were told that the ofitemscompletedcorrectlyandthenumberof
PCATD aircraft was not programmed for any errorsforeachofthesixflightsegmentsforthat
system failures and that the flight pattern was a particular flight. As in the baseline phase, the
radar-vectored instrument flight, with an in- experimenter printed out a technical flight
strumentlandingsystemapproachtoafull-stop diagram as well. The experimenter then entered
landing. They were also told that their behavior the room. He showed the technical flight
during the flight would be observed and diagram to the participant first and discussed
recorded using Web cameras. They were then the technical merits of the flight. He then
shown the flight checklist and asked to use it as showedthetwochecklistfeedbackgraphstothe
they would during regular flights. In addition, participant and praised any improvements.
they were told that the experimenter would Praise was provided only following improve-
provide them with some postflight information ments. This protocol was repeated for each
after each flight and that it would take him flight.
about3to5 mintopreparethatmaterial.They Withdrawal of treatment. Feedback was no
would thus have a short break after each flight. longer provided for use of the flight checklist
Although this break was not necessary to after each flight. This phase was identical to the
provide the postflight technical information baseline phase.
duringthisphase,itwasnecessarytopermitthe
experimenter to summarize the participant’s Interobserver Agreement
checklist performance during the intervention For each trial, either the first or third author
phase. Thus, the same postflight break was was the experimenter or primary observer. The
502 WILLIAM G. RANTZ et al.
other then independently watched randomly RESULTS
selected recordings of the flights and scored
Figure 1 displays the total number of
performance using the checklist observation
checklist items completed correctly for each
form. After a participant completed the study,
participant. All participants increased perfor-
numbers corresponding to each trial were
mance accuracy over baseline when postflight
placed in a container and 10% were randomly
checklist feedback was provided, and improve-
drawn.Carewasinsuredthatatleastonesession
ments were maintained after intervention
during each condition was selected for each
withdrawal.
participant. This ensured that (a) 10% of the
Baseline checklist performance varied con-
sessions were rescored for each participant and
siderably among participants, with P1 showing
(b) the trials that were rescored were randomly
the highest level of performance and P2
selected, with the restriction that at least one
showing the lowest level. Baseline trends were
session during each condition was rescored.
stable over time with the exception of P3, who
Interobserver agreement was calculated sepa-
showed a sudden increase in accuracy following
rately for the occurrence of correct checklist
the first two trials, and P6, who showeda slight
behavior and for the occurrence of errors.
downward trend.
Agreement on the occurrence of correct check-
Following the introduction of feedback, 5
list behaviors was computed by dividing the
participants(P1,P5,P6,P7,andP8)showedan
number of agreements on the occurrence of
abrupt change in the number of checklist items
each checklist behavior completed correctly by
completed correctly, 2 (P2 and P4) showed a
the number of agreements on the occurrence of
level change followed by an increasing trend,
eachchecklistbehaviorcompletedcorrectlyplus
and 1(P3) showeda graduallyincreasing trend.
the number of disagreements. Interobserver
Overall the mean number of checklist items
agreement on the occurrence of the percentage
completed correctly for all participants in-
of checklist errors was computed by dividing
creased from 22.2 during baseline to 39.2 (of
thenumberofagreementsonchecklisterrorsby
40) during the last three sessions of the
the number of agreements on checklist errors
intervention phase.
plus the number of disagreements. Mean
Each participant maintained high levels of
agreementontheoccurrenceofcorrectchecklist
correct item completion after the graphic
use was 90% (range, 61% to 100%). Mean
feedback intervention was withdrawn. The
agreement on the occurrence of errors was 82%
(range, 0% to 100%). The occasion on which mean number of checklist items completed
mean agreement on errors was 0% was for a correctly was 39.6 (of 40) after the withdrawal
session in which one observer scored one error of treatment for P1, P2, P3, P4, P5, and P6.
and the other observer did not score any errors. Because P7 and P8 were not able to attend as
many sessions as the others, they were not able
Integrity of the Independent Variable to move to the withdrawal condition before the
To be sure that the technical flight and end of the semester.
checklist feedback was administered correctly, The percentage of total checklist errors for
the experimenter read from prepared scripts. In each flight segment for each participant during
addition, participants were asked to initial the each condition isshownin Figure 2. Duringall
technical flight diagrams and the checklist flights, 1,973 total errors were observed. The
feedback graphs that were used during post- percentageoftotalchecklisterrorswashighand
flight briefing sessions and give them back to variable during baseline. The mean percentage
theexperimenter.Participantsinitialed100%of of checklist segment errors was highest for the
all flight diagrams and feedback graphs. after-takeoff checklist (88%, range, 71% to
FEEDBACK AND CHECKLIST COMPLETION 503
Figure 1. Total number of checklist items completed correctly by each participant during each session of
theexperiment.
504 WILLIAM G. RANTZ et al.
Figure2. Percentage oferrors for each condition perflight segment.
FEEDBACK AND CHECKLIST COMPLETION 505
100%) and lowest for the pretakeoff portion of been provided for desired work behaviors
the flights (32%, range, 6% to 69%). (Austin et al., 2005; Crowell et al., 1988; Wilk
For all participants, errors decreased or were & Redmon, 1998). The present research is
eliminated during intervention. During with- novel, however, in demonstrating such effects
drawaloftreatment,3participants(P1,P5,and for the use of flight checklists by pilots in an
P6) performed perfectly. P2, P3, and P4 had extremely challenging and dynamic situation.
near-perfect performance. Although this study was conducted in a
Figure 3 shows the mean percentage of total simulated environment, the results suggest that
errors per checklist item across individuals graphic feedback and praise could be used to
during baseline. Percentages that are 50% or increasetheextenttowhichpilotsusechecklists
greaterareshadedforeachparticipant.Also,the accurately, potentially preventing crashes and
checklist item name is shaded if the percentage saving lives. A large number of studies show
of error was 50% or greater for 4 or more that flight skills acquired in the simulator
participants.Threechecklistsegmentsemergeas transfer to the aircraft; however, none of these
having the highest percentage of errors. The studies has specifically examined whether
highestfrequency of errorsoccurredforthetwo checklist use taught in a simulator transfers to
items in the after-takeoff segment: checking the flight deck of the aircraft. Further research
flaps and engine instruments, with 99 errors should focus on installing data loggers and
each. These errors occurred on approximately cameras on the aircraft to measure checklist use
50% of the total flights across all participants. before and after training. If checklist use did
The prelanding items were the next problem- generalize, it would have majorimplications for
atic. Six of the 8 participants (P1, P2, P3, P4, simulatedflighttraining,whichislessexpensive
P5, and P7) had high percentages of errors on and less risky than in-flight training. Even if
all five items in this segment. The after-landing checklist use did not improve without adding
segment had the third highest errors, with 4 enhancements to promote generalization, these
participants (P2, P3, P5, and P7) having high results would be directly applicable to the
error rates on all four items. operators of UAVs, for whom actual flight is
indistinguishable from simulation.
Baseline performance varied considerably
DISCUSSION
acrossparticipants.Moreover,someparticipants
The use of paper checklists in the flight performed very poorly. The variables that
environment remains a vital component to safe contributed to poor baseline performance are
operations. As with the airline audits conducted not known but may be due to (a) poor initial
by Helmreich et al. (1999, 2001) and Klinect, flighttraining,(b)noorinfrequentfeedbackon
Murray, Merritt, and Helmreich (2003), the checklist use during training or nontraining
resultsofthecurrentstudyrevealedthatchecklist flights, (c) no aversive consequences for failing
compliancevariedconsiderablyacrossindividuals to use the checklist accurately in the simulated
duringbaseline,ranging from amean of 53%to setting (e.g., no emergencies that could lead to
91%. When the intervention was introduced, actual danger, no crashes possible), or (d) a
accuracy increased rapidly to near-perfect levels combination of these variables.
for each participant. Furthermore, those changes Five participants (P1, P5, P6, P7, and P8)
were maintained after the intervention was showed an abrupt change in performance after
removed, continuing for seven simulated flights. the first intervention trial. In addition, all 5
Theincreasesafter intervention aresimilar to maintained high levels of performance after the
increases that have been reported in other withdrawaloftreatment.Theabruptincreasein
studies when graphic feedback and praise have accuracy after one intervention session and the
506 WILLIAM G. RANTZ et al.
Figure3. Percentageoferrorsperchecklistitemacrossindividualsduringbaseline.Itemsareshadedifaparticipant
missedthatitem50%ormoreofthetime,andchecklistitemsareshadedif4ormoreparticipantsmissedthatitem50%
ormore ofthe time.