NASA Technical Reports Server (NTRS) 20030013637: Graphical Weather Information System Evaluation: Usability, Perceived Utility, and Preferences from General Aviation Pilots

20 Pages·1.9 MB·English

by  NASA Technical Reports Server (NTRS)

Paper Number 2002-01-1521
Graphical Weather Information System Evaluation: Usability
Perceived Utility and Preferences from General Aviation Pilots
Kara A. Latorella
Crew/Vehicle Integration Branch, NASA Langley Research Center, Hampton, VA 23681
James P. Chamberlain
Crew Systems and Operations Branch, NASA Langley Research Center, Hampton, VA 23681
ThisPaperisaworkoftheU.S.GovernmentandisnotsubjecttocopyrightprotectionintheUnitedStates.
ABSTRACT these accidents, 66%, resulted in fatalities. Convective
weather is challenging because it can include
Weather is a significant factor in General Aviation (GA) severe/extreme turbulence, gusts, hail, icing, lightning,
accidents and fatality rates. Graphical Weather reduced ceiling and visibility, instrument meteorological
Information Systems (GWISs) for the flight deck are conditions (IMC), and possibly severe downdrafts and
appropriate technologies for mitigating the difficulties GA microbursts. Such concomitant weather phenomena were
pilots have with current aviation weather information analyzed separately in the AOPA accident analysis.
sources. This paper describes usability evaluations of a Therefore the incidence of GA accidents attributed to
prototype GWIS by 12 GA pilots after using the system in convective activity, and the fatalities resulting from such
flights towards convective weather. We provide design weather systems is likely under-represented by the
guidance for GWISs and discuss further research required percentages cited for only thunderstorm effects.
to support weather situation awareness and in-flight
decision making for GA pilots. GENERAL AVIATION WEATHER INFORMATION
Today's pilots of small GA aircraft principally rely on aural
sources and external, or "out-the-window," weather cues
for weather information. Aural sources can include direct
INTRODUCTION queries to Flight Service Station (FSS), En Route Flight
Advisory Service (EFAS, or "Flight Watch"), and Air Traffic
GENERAL AVIATION WEATHER ACCIDENTS - Eighty- Control (ATC) personnel, as well as monitoring these radio
five percent of the aviation accidents that occurred from frequencies to overhear other pilots' comments, queries,
1990-1996, and nearly eighty-five percent of the accident and the information supplied to them by ground-based
fatalities, involved small GA airplanes. One major professionals. Pilots can also tune in automated weather
contributor to aviation accidents is hazardous weather. information services such as HIWAS, AWOS/ASOS, and
This equates to, on average, eleven weather-related GA ATIS to obtain a broadcast of conditions over a large area
accidents per week. Desktop simulation, and other or at specific reporting stations. Unfortunately, the
information available from broadcast aural sources is
laboratory experiments demonstrate pilot errors that
corroborate the implication of this accident statistic limited and, when weather becomes a problem, the
[1,2,3,4,5,6]. Following the 1997 Gore commission on frequencies used to obtain "live" aural information become
Aviation Safety, NASA initiated the Aviation Safety saturated, making this information inaccessible at exactly
program (AvSP) with the goals of reducing the aircraft the time it is most needed. When asking pilots to
accident rate by a factor of 5 within 10 years and by a describe their current communication method for obtaining
factor of 10 within 25 years. Within AvSP, the Aviation weather information during GA flights, 78.4% of the
Weather Information (AWIN) program element aims to answers relied on radio communications [8]. Lack of
contribute to these goals by improving the weather clarity in (35%), and loss of (29%) these radio
information available to aviation users. communications were most prominently mentioned as
limitations on the effectiveness of this method for
General aviation is particularly affected by convective obtaining weather information [8].
weather. A survey of GA accidents from 1982 to 1993 [7]
revealed that while only 3.5% of these accidents were Currently, pilots of small GA aircraft have limited in-
directly attributed to thunderstorms, a large percentage of flight information about convective weather activity,
especially when compared to that available on larger
aircraft.Unlikelargeraircraftm, ostsmalGl Aaircrafatre
sensors or observations; if uplinked, what the size of the
notequippewdithonboarwdeathedretectioenquipment
packets are, the form of communication messaging
suchas onboardweatherradaror lightningdetection
(broadcast, request/reply), and the rate of uplinking new
systems(e.g.,StormscopeStrikefinder that can information. The content of the information available on a
TM, TM)
indicate convective activity. Onboard weather radar GWlS must also be selected to support flight decisions
systems that are available for small GA aircraft are and support a pilot's ability to assess the reliability of the
typically expensive, and limited in performance by size information available. The manner in which this weather
and power constraints. When available, these systems information is presented should facilitate interpretation of
can provide improved weather awareness for severe the location and intensity of weather phenomena, assist
weather hazards, but are limited in range and accuracy pilots in determining the relevance of weather to the
[9]. Onboard weather radar systems are workload- mission, and assist pilots in projecting the location and
intensive to use accurately [10], are subject to intensity of weather phenomena over time. GWlSs may
attenuation, have a limited range, and provide information also include aiding functionality to assist pilots in
that is primarily forward of the aircraft and at the aircraft's intelligently acquiring relevant weather information,
altitude [11]. While these systems show severe local determine when weather presents a hazard to the
weather to avoid, they do not provide the more mission, and suggest or evaluate actions in response to
comprehensive weather picture required to fully support hazards detected. Finally, the more mundane
strategic planning or avoidance maneuvers. More considerations of human-computer interaction must also
accessible, complete, and usable weather information be well-designed to result in a usable system. These
would benefit pilots' situation awareness, decision- considerations include selection of usable input devices,
making, and safety. Graphical presentation is a more legible fonts, easily navigable menu structures,
appropriate representation for this type of information [12], appropriate screen resolutions, timely system response to
can more effectively be integrated with other flight inputs, meaningful coding and symbols. Finally, the
information (e.g., terrain) and can be extended using system must be robust to the ambient conditions existing
symbols. inaviation, and therefore be usable in turbulent conditions,
as well as in direct sunlight and at night. Considering the
The FAA Flight Information Services Data Link breadth of human factors concerns in designing a GWlS,
(FISDL) program will soon make data-linked weather there is a dearth of research that directly addresses these
information systems widely available to GA pilots via concerns.
commercial FISDL vendors. FISDL vendors provide, for no
service charge, uplink of textual aviation weather products, Principally, prior research focuses on simply
including weather observations (METARS & SPECIs) and answering the question "Does graphical weather
forecasts (TAFS) of terminal environments, as well as information improve pilot decision making?" As one might
reports of severe weather conditions (SIGMETS, expect, based on theory, access to graphical weather
Convective SIGMETS, AIRMETS, and severe weather information can assist pilots. Pilots using a prototype
forecast alerts) and pilot reports (PIREPS). For a fee, GA GWlS in static and dynamic desktop and flight simulation
pilots may augment this basic information. One of the experiments were shown to be more likely to acquire
first available graphical products is a national weather trend data, have a more comprehensive awareness [4],
radar mosaic (NEXRAD). The FISDL textual and graphical make better go/ no-go decisions, rate hazard levels
weather information is broadcast by a network of VHF higher, have more confidence in weather-related
ground stations, and received and displayed by an decisions, make fewer calls to ground aviation weather
onboard GWlS. NavRadio Corporation (now part of the personnel [13, 14], make more correct decisions with
Bendix-King Division of Honeywell International), in a graphical weather information than with either verbal or
cooperative agreement with NASA AWlN, developed the text alerts [15, 16], and use 5% less fuel and clear
prototype GWlS used in this study and was subsequently thunderstorms by 3 times the margin of pilots without a
selected as one of the two FISDL program participants. GWlS [17]. When used in a GA flight test, accompanied
The other FISDL participant is ARNAV Systems, Inc. by terminal forecasts and surface observations, and
integrated with a traffic information service, subjects
commented enthusiastically on the utility of a GWlS [18].
GWIS ASSESSMENTS - The design of a GWIS involves More than 82% of subjects had positive responses to the
many human factors considerations. The entire system utility of precipitation maps, surface observations, and
must be designed such that information is available to the terminal forecasts individually [19]. All subjects had a
user that is reliable and temporally and spatially relevant positive overall impression of the system; 88% indicating
to the decisions it supports. These considerations that it would be important to make available to GA
determine the requisite datalink capability, and onboard operations [19]. Pilots using a GWlS in two-person crews
sensors and processing units, and constrain the design of in a commercial flight deck environment also
the information infrastructure that supports the GWlS enthusiastically embrace this technology [17, 20].
display. Decision requirements therefore should be used
to determine what information is best acquired through FAA FISDL and NASA AWlN jointly funded a
onboard sensors, data-link equipment, ground-based simulation experiment at Research Triangle Institute (RTI)
toevaluatepilotweathefrlyingwithandwithouat GWIS andindicatetshatbenchmartkasksaremoreor less
similartotheprototypeevaluateidnthispaper.Inthis easilyaccomplisheind,largepart,duetotheusabilitoyf
study,subjectwsereinIMC,hadaccesstoanautopilot, these devicesand menunavigation[28]. This
andwerenotgivena presenptositionsymbool nthe investigatioanlsoshowedthatsomeofthesesystems
GWISdisplay.ResultsindicatedthatwhilethisGWIS provideautomaticbrightnesscontrolto improve
increaseadwarenesosfthegeneralol catioonfconvective readabilitwy,hileothersallowfornoopportunittoycontrol
weatherit,didnotimprovepilotdiversiodnecision-making brightnes[s28],aseriousdetrimentot useinanactual
(subjectdsidnotunderstanthdelocationofweathewrith aviatioennvironment.
respecttotheirposition)in,creasewdorkloafdoratleast
halfthesubjectsa,ndreducedrelianceonground-based Insummartyh,erearemanyaspectosfGWISdesign
weatheprrofessiona[2ls1,22].So,whiledemonstrating that,if improperliymplementedc,ouldresultin pilots
positiveeffectsinasimulatioennvironmeannt,dbeingwell havinganincorrecatnd/oirncompleutenderstandionfgthe
receivedbypilotsin a demonstratioint,appearsthat weathera,swellasincreasewdorkloaadndanexcessive
GWISsmaynotuniversalilmyproveflighdt ecisions. opportunictyosttootherflightdeckduties.Thesedesign
considerationisnclude,but are not limitedto: the
Priorresearchevaluatingpilot performancweith informatiotonbedisplaye(dtypeofweatheirnformation,
GWISsindicatessomespecifidcesigndecisiontshatwill geographicraelferences,
etc.), the manner in which this
influencethe usabilityand utilityof thesesystems.
information is presented (color coding, resolution, level of
Participantsin a two-crewsimulationexperiment
abstraction, symbology, etc.), and the characteristics of
suggestetdhatwhileradarsummarayndlightninsgtrike
the physical interface that houses this system (input
informatiownas critical,theyonlyneededeitheran
devices, brightness control, etc.). Rather than suspecting
IFRNFRcategormy apora pagethatshowedsymbols
that GWISs in general are not appropriate technologies,
codingsurfaceobservatiocneilingandvisibilitfyorstations
we suggest that more work must be done on designing
onamap[17].Therewasnocleardeterminatiwonhichof
the information provided by these systems to be reliable,
thesetworedundanintformatiosnourceswaspreferred
action-oriented [29] and decision-centered, and on
[17]. TheRTIstudythatfailedtoshowa significant
designing the interface to these systems to be more
advantagoefa GWISu' seonaviationdecision-making
easily and effectively used.
attributedthis resultto pilots'failureto understand
weatherlocationwithrespecttotheiraircraft'sI_sition
[21,22].Pilotcomplainctsorroboratethdisinterpretation THE COWS EXPERIMENT - The AWIN Convective
[23]. A subsequenetxperimentw, hile failingto
Weather Sources (COWS) experiment investigates how
demonstraateperformancaedvantagein,dicatetdhatan
GA pilots use weather information available from aural,
own-shippositionsymbolreducedpilotworkload[23].
"out-the-window" visual, and GWIS-displayed cues, to
ThisexperimenatlsodemonstratethdatNEXRADdata
support in-flight decisions related to convective weather
resolutioinmpacteaddecisiotnhatrequireedstimatioonf
systems. While the focus of this research is to better
distanceto a stormcell;thatis, pilotswith larger
understand how GWISs are used in in-flight decision-
resolutionNEXRADdata (8kmsquare)madesafer
making, the experimental protocol also allowed us to
decisiontshanthosewithsmallerresolutiodnata(4km
conduct a usability assessment of the prototype GWIS
square)[23]. Oneearlyimplementatioofnanuplinked
implementation we used. Two earlier publications
radarmosaicGWISd,evelopeadtMITLincolnLabswith
described preliminary CoWS results based on partial data
fundingfromtheFAADatalinOkperationRalequirements
collection. The first [30] discussed pilots' relative
Team(DLORTh),ada15minuteupdaterate,6km-square
confidence, information sufficiency, and workload ratings
resolutioanndemployead"lossya"lgorithm(resultingin
when using aural, out-the-window visual, and graphically
lesswell-definepdrecipitatioanreas)tocompensaftoer
represented weather information cues in flight near
loweravailablebandwidt(h250bps[2)4,25]. Indesktop
convective weather. The second discussed the accuracy
usabilitayssessmenatsll,subjectfsoundthehighleveol f
and consistency of the test subjects' ability to identify
Iossycompressiounnacceptablaen,dsomefoundthat convective weather relative to their aircraft location and
themediumlevellackedthefunctioneaql uivalencoefthe
flight track [31]. This paper discusses the results of the
uncompressiemdage[14,26]. Priorresearchhasalso
usability assessment and participants' comments on
suggestedthat improvemenatsre not limitedto the
usability and utility of this GWIS.
presentatioonfgraphicawleatheirnformatiosnu,ggesting
thatMETARinformatiosnhouldbepresentedin plain
Englishtranslationr,atherthanin the standardICAO OBJECTIVE
teletypeencodin[g23,27].Priorresearcshuggesttshat
earlyGWISprototyperseducepilots'interactionwsith
The International Standards Organization has defined the
FSSandATC,andtherebymayreducetheoveralvliew
"usability of a product (as) the extent to which the product
theyhaveontheweather[21,22]. A humanfactors
can be used by specified users to achieve specified goals
evaluatioonfseveraml ulti-functioncaolckpistystemwsith
with effectiveness, efficiency, and satisfaction, in a
GPSmovingmapoverlaysd,emonstrattehsevarietyof
specified context of use" [32]. In the CoWS experiment,
inputdevicesavailableforcockpiitnformatiosnystems,
GA pilotswereaskedto assessthe usabilityof a
the radar product (upper left corner), a scale legend (upper
prototypegraphicawl eatherinformatiosnystemin the
right corner), and indicates missing data (horizontal yellow
contexotfa flightinthevicinityofconvectivweeather.
stripe where data was missing). This graphical information
Whileparticipanwtserenotflyingtheaircraftp,articipants
could be viewed alone, or augmented using three other
ratedflightscenariosv'alidityas veryhighandthey
modes of information presentation: NEXRAD Mosaic,
consideredthese experimentafllightsto be fairly
METAR, or both NEXRAD and METAR (NEX/MTR). When
representativien, termsof informationavailableand the METAR information was available, the text for selected
workloado,ftheirtypicafllights[30].Theobjectivoefthis
METAR icons was available. When in NEXRAD-only or
paperis topresenttheusabilitydataobtainedforthis
Graphics modes, the identifiers for airports or NAVAIDs
prototypeGWlS,by this userpopulationandin this were available. The user could select one of three GPS
contexta,nddiscusstheimplicationosftheseresultsfor
modes: GPS Off (no aircraft position/track symbol), GPS
improvintghedesignofGWlSs.
Lock (display centered on the aircraft's position), and GPS
Free (aircraft symbol provided, but the display was not
locked to this position). The joystick could be used in
METHODS three modes: Scroll (to view that which is off the edges of
the displayed map; not usable in GPS Lock mode),
APPARATUS Apparatus for the CoWS experiment Crosshairs (to select METAR, Airport, or NAVAID
included supporting ground infrastructure, test aircraft, and symbols), and Zoom (to change map scales: 500, 200,
the tethered GWIS. Four prototype AWIN/Honeywell 100, 50, 25, 10 nautical miles per 1.5cm, or 1/7 th, of
broadcast VHF data link (VDL) transmitters were located display width and 1/5th of display height). At the 500nm
in Virginia and provided a broadcast link of packaged scale, the user could see the map for the entire
weather data files to the test aircraft along four routes of continental United States. The interaction and
flight (Figure 1, rings indicate 40nm broadcast range). informational elements of this GWlS are described in more
detail with reference to results in a following section.
Figure 1. CoWS Experimental Test Range.
NASA Langley's Raytheon B-200 Super King Air, a nine-
passenger, pressurized twin-turboprop airplane, was
operated at speeds and altitudes consistent with those of
the smaller, piston-engine GA aircraft used by the Figure 2 - The CoWS Prototype GWIS.
participant population. The onboard GWlS included aVDL
receiver, Global Positioning System (GPS) receiver, and
SCENARIOS - The ideal flight scenario operated under
two laptop PC's with tether cables to two small handheld
display units. The display unit screens were approximately Instrument Flight Rules (IFR) but in Visual Meteorological
Conditions (VMC). The test aircraft departed from NASA
7.5cm tall by 10.5cm wide. Five bezel buttons (12mm x
6mm, each) on the right side of the unit actuated soft Langley/ Langley Air Force Base (LFI) on a flight path
that, if allowed to continue, would obliquely intercept a
menu fields, and a rate-controlled joystick controls pan,
zoom, and crosshairs for symbol selection (Figure 2). The frontal convective system of at least moderate intensity at
approximately 120nm from top-of-climb, and at an altitude
unit presented Iossless, nationwide radar mosaic imagery
at 4-square-km resolution with a 6 minute nominal update above the haze layer (typically 14,000 feet). The location
of the GWIS' ground-based infrastructure and other
rate assuming adequate broadcast reception, and surface
weather observations (METARs) in text and symbolic form airspace considerations constrained the region in which
these flights could occur. To accommodate this
for reporting stations in the mid-Atlantic region. The
display also presented contextual features (rivers, constraint and minimize training and materials, four
potential IFR flight plans were developed from LFI to
interstates, and state boundaries), airport identifiers,
present position and track symbol, creation time stamp for Hickory, NC [HKY]; Charleston, WV [CRW]; Abingdon,
VA [VJI]; or Clarksburg, WV [CKB]. One of these four
flightplanswaschosenonthemorningofeachflight
participant from each of the clusters to balance exposure
basedonprevailinwgeathecronditionsP.articipandtsid
experience across flight scenarios. Cross-country
notperformflyingdutiesduringtheseflightsa; NASAtest
experience level ranged within teams and was counter-
pilotservedaspilotincomman(dPIC).Theconvective
balanced over teams to mitigate concerns about
weatherduringthe flighttests couldgenerallybe
generalization to the participant population and
describedaswell-definesdi,gnificanlitnesandareasof
(experience x flight) interactions. Cue assignment to
cells,withsurroundintgoweringcumulusbuildupsand
participant experience levels was counter-balanced to
occasionaelmbeddedcells. Flightconditionswere
mitigate concern about (cue x experience) level
generallyunrestricteidn visibilityo,ntopofanylower interactions.
cloudlayersa,ndlaterallcylearoftoweringcumulusand
cumulonimbucsells. Weattemptedto achievethese
weatherconditionosnthe outboundlegof allflights.
PROTOCOL - When a participant team arrived at NASA
Duringtheinboundportionoftheflightsw,etypicallytried
Langley inthe morning, each participant was provided with
toflyascloseas20nmtointerestincgonvectivceells.
an introductory briefing, consent form, schedule, and
Preliminary Questionnaire. They then received a mission
motivation and briefing; a local terrain, NAVAID and airport
EXPERIMENTCAOLNDITIONASNDDESIGN The
identifier review; a route briefing for the flight to be taken;
experimenctollecteddataaccordintgoawithin-subjects
and practice on forms and procedures to be used during
experimentdaelsign.Threepilots,constitutinagteam,
the outbound experimental phase. Following a short
participateindeachflight.Theteamflewthreetimes,
break, participants had 40 minutes in total, to review a
allowingeachparticipantotexperienceeachexperimental
standard preflight weather briefing composed of a DUATS
condition.Oneachflighto,neparticipanwtasallowedto
text briefing, associated weather graphics, and a pre-
use the GWIS,and also receivedaural weather
recorded briefing from a Flight Services professional.
informatiodnuringtheoutbounedxperimenptahlaseofthe
Participants then completed the Preflight Weather
flight.Thisparticipancot mpleteadusabilitsyurveyduring
Situation Awareness questionnaire. While other
theinboundphase.Oneoftheothertwoparticipants
participants completed knowledge tests and personality
receivedanout-the-windovwiewaswellas theaural
inventories, the participant who was assigned to receive
informationT.heotherparticipanretceiveodnlytheaural
the AWIN GWIS was trained on the display. This training
weatherinformation.Participantwshohadusedthe
used a scripted Microsoft Office Powerpoint
TM
GWISonapriorflightwereallowedtolookattheGWIS
presentation with digital photographs of the actual screens
brieflyduringtheinboundphase. Theauralweather
and a scripted aural instruction to introduce the GWIS.
informationincludedlisteningto a HIWASbroadcast
This training device was designed to allow participants to
stationg,ettingaFlighWt atchbriefingandgettinganATC
interact with a representation of the actual display, albeit
report. Theaural-onlyconditionrepresentswhatis
along the lines of the script, and to provide a standardized
availableto mostGApilotsduringIMCtoday. The
training. The training explained the interface control
conditionwithauralinformatioanndtheout-the-window
features and modes; and described the information
view representswhat GA pilotshavefor weather
presented by the system, including symbols and color-
informatioinn today'sVMC. Theconditionwithaural
coding. Following this standardized training, participants
informatioanndthe GWISrepresentwshatGApilots
received a comprehension survey. This survey required
mighhtaveinthenearfutureinIMC.
participants to interpret screen symbols and color
conventions, locate information regarding age of weather
data, demonstrate knowledge of the menu structure and
PARTICIPAN-TPSarticipanwtsererecruitedfromlocal
display modes. The results of this survey were used to
regionaalirportsandthroughadvertisemenAt.pplicants
indicate requisite compensatory training. Usually only a
reportetdheirflyingexperiencaendweatheerxposuroen
few items, if any, required this compensatory training.
a BackgroundQuestionnaire.Participanstelection
Participants were able to explore the actual system during
criteriaincludeda:ninstrumenratting1, 0-50flightt-ours
ground operations and during ascent once aboard the test
withinthelast90daysa,nd50-100c0ross-counotrry100-
aircraft, and were given a quick review by an experimenter.
2000totalflighthours.Inadditionp,articipanwtserenot
selectewdhohadworkedforascheduleadir-carrieinrthe
The in-flight portion of the experiment began after the
prioryearorwhohadparticipateindtheaforementioned
aircraft had climbed to cruising altitude and when the
RTI/AWINexperiment.Weatheerxperiencheasbeen
aircraft was approximately 120nm from the first convective
foundto significantlayffectweather-relateddecision-
weather area of moderate or greater intensity. The
makingandinformatioancquisitio(nWiggins& O'Hare
outbound leg of the in-flight portion concluded when
1995)so candidateparticipanwtsereclusteredinto3
approximately 20nm from this area, or at approximately
groupsof "exposureexperienceu"singcross-country
100nm from the initial experiment starting point, whichever
hours.Themidpoinotsfeachclustearre135(low),379
occurred first. Throughout the outbound phase of the
(mediuma),nd738(highc)ross-counthroyursrespectively
flight, Weather Situation Awareness (WXSA)
(p<
.0001). Twelve participants were selected to form questionnaires were given every 8 minutes (approximately
four three-member teams, each team composed of one every 25nm), and Position Update tasks and aural
weatheirnformatiownerealternateplyrovidebdetweetnhe RESULTS
WXSAquestionnairessu,chthateachwas provided
approximateelvyery16minutesE. achoftheseeventsis
Participants' responses on the usability, debriefing, and
describebdelow.
inbound questionnaires are reported below with
annotations from extensive debriefing sessions. These
The PositionUpdatetask was designedto
data reveal users' perceptions on the general usability of
compensatfeorthe lossof positionaalwarenesasnd
this GWIS, usability of interaction elements, and usability
workloadinducedby not piloting. For this task,
and utility of information elements. Protocols from
participantcsopiedschedulerdeportsfromthepilotin
debriefing sessions provide the basis for discussions of
command(airspeeda,ltitude,heading,position,next
how these participants would use this GWIS, how use of
waypoinat,ndcurrenttime)ontoapreparefdorm;plotted
this system would affect situation awareness, flight deck
positionon an IFRlowaltitudeen routechart;and
decision-making, collaboration among NAS users, and
calculatedelapsedtimeandgroundspeed.Theywere
pilot workload. One-tailed t-tests were conducted on
alsorequiredtonoteanyATCtransmissioanffectintghe
rating scale data to test if values were significantly (c_=
flight.Allparticipanrtesceivedscheduleadurawl eather
.05) greater than 50% of the scale, and greater than 75%
informationT.hefirstauracl uewasobtainefdromalocal
of the scale. For the remainder of this document, where t-
automateHdazardouIsn-FlighWt eatheArdvisorSyervice
test statistics are significant for means greater than 75%
(HIWASb)roadcasotutlet,the secondfromquerying
of the scale, the adjective "very" is used to describe the
EFASpersonnealn, dthethirdfromqueryinAgTC.The
average rating response. For t-test statistics significant
WXSAquestionnairweserehandedtoparticipanatstthe
for means greater than 50%, but not significant for means
indicatetdimes.TheWXSAitemsaddressepdarticipants'
greater than 75%, the adjectives "fairly" and "marginally"
weathersituationawarenesasndflightdecisions.In
are used to describe the average rating response.
particulapr,articipanwtsereaskedtoidentifythelocation
Because debriefing sessions were relatively unstructured,
of the nearestconvectivecells. Participantwsere
numbers of participants cited as commenting on a
instructedthattherewouldonlybe enoughtimeto
particular aspect of the system should be interpreted as
completethe WXSAif they rely on their"mental
indicative of the salience of this issue to participants, but
snapshoto"f theweathertheyhaveat thetimethe
not necessarily a percent agreement of the sample
questionnaiirseadministered.
population.
Attheconclusioonftheoutboundleg,participants
wereaskedtoplottheaircraft'spositionontheirenroute
GENERAL ASSESSMENT - Generally, participants found
IFRchartd,rawweathewrithin50nmoftheflightpathon
the functionality of the GWIS fairly adequate. In
thecharta,ndcompletteheInboundQuestionnairTeh.is
particular, they were enthusiastic about the advantage of
instrumenctontainedNASA-TL[X34]-derivedscalesfor
having graphical NEXRAD weather in flight, to the point
workloadassessmenat,skedparticipanttso indicate
that other interface deficiencies appeared relatively
otherweathesrourcetshatwouldhavebeenhelpfula,nd
insignificant to them. Participants' ratings, on average,
abouttheirflightdecisionsA. ftercompletinthgeInbound
indicate that they were fairly comfortable in describing
Questionnaitrhee,participanutsingtheGWISwasasked
their interaction with the system as more "wonderful," than
to completea UsabilityQuestionnairweh, ichincluded
"terrible;" and more "satisfying," than "frustrating." They
someQUIS[35]items,andprovideanyadditional
were very much more willing to describe their experience
commenthsehad.Followintgheflight,participanwtsere
with this system as "easy," than "difficult." Participants'
providedwitha shortdebriefingquestionnaifroerthat
comments regarding the general usability of this GWIS
flight.Attheconclusioonfthethirdflighftorateamw, hen
were fairly positive, and generally reflected that the utility
all participantshad beenexposedto the display,
of having a graphical weather information system on board
participantasnd experimentermsorefullydiscussed
outweighed any specific concerns they had with the
issuesof experimentavlalidityanddisplayusability.
interface. Rasmussen and Vicente [36] emphasize the
These final debriefingsessionstypically lasted
importance of designing systems such that they invite and
approximatetlwyohours.Datausedinthispaperis
are robust to exploration and learning in the operational
extractepdrincipallfyromtheUsabilitQyuestionnaairned
environment when possible. The ability to easily learn and
notesfromfinaldebriefinignterviewsS.electeedlements
explore the system is therefore important aspects of
oftheWXSASurveyandtheDebriefinQguestionnaire
general satisfaction with this system. Participants'
augmenttheseprincipaslourceswheretheyaddress
ratings associated with the ease of learning and exploring
relatedissues.
this system were also positive. On average, ratings
indicated that the participants thought that learning the
system and advanced features was fairly easy, getting
started with the system was very easy, and the time to
learn to use the system was fairly fast. On average,
ratings indicated that the system supported participants'
need to explore features; that the system was very
encouraginogfthisexploratiotnh,atitwasverysafeto
label and size of areference bar that is in the upper right
explorefeaturesa,ndthatdiscoverinngewfeaturewsas
corner of the display. The area displayed is 7 times the
fairlyeasy.Participangtsenerallsytatedthattheyhada
width and 5 times height of this scale bar when the menu
comfortabflaemiliaritwyiththesystemafterdesktopand
labels are off. Participants were asked to indicate the
preflighttraining. Oneparticipanfteltthatadditional
scales they used during the experiment. The distribution
trainingwasstillnecessartyousethesystemeffectively
of these responses (Figure 3) indicates that these data
andcommentethdatheexperienceindformatioonverload
are fairly normally distributed around the most popular
withthesystem.
50nm scale. During debriefings, some participants
mentioned the scale they would select ifthe GWIS did not
Aswedetailbelow,howevert,heenthusiasmwith
allow them to change scales. Two participants favored
whichparticipantgsenerallyregardedthe systemand the 100nm scale, but one of these noted that he would
theirabilityto startusingit is lessevidenwt henthey want that to be smaller if there were weather in the
respondto questionsaboutparticulaarspectsof the
vicinity; one responded 50nm, and one 25nm. The 50nm
system.Wepresenttheirresponseasccordintgothe
scale is the largest scale that includes all the contextual
aspectsoftheGWISthatareusedtomanipulattehe
and aviation location information. Two participants
informatiosnhownt,he
Interface Control Elements, and indicated that also having a scale that shows 2 to 5nm
the information that is displayed on the GWIS and how resolution would aid ground movement.
this information is displayed, i.e., Presentation and
Information Elements.
14
INTERFACE CONTROL ELEMENTS The physical
-_ 12
interface of this system includes a knob for adjusting
brightness, a small rate-controlled joystick, and five bezel
keys. Participants were not explicitly asked about use of
the brightness knob, but few were observed to use it.
_ 6
Only one participant expressed displeasure with the _- 4
joystick control, finding it too sensitive. The bezel keys
2
determine the mode of the joystick, the type of weather
information displayed, how GPS position information is 0
used, and whether the soft labels associated with these lOnm 25nm 50nm lOOnm 200nm 500nm
keys are shown. One of these keys also, depending on
the type of weather information displayed, allows access
to METAR text information and enables labeling of Figure 3. Reported Frequency of Map Scales Used
selected airport and NAVAID symbols. Participants did
not comment on the physical interface of the bezel keys.
Weather information modes can be either "Graphics" (no
weather information, but all contextual features, and The Crosshairs mode allows users, with the joystick,
access to airport and NAVAID identifiers), "METAR" to move a vertical and a horizontal line to select an airport
(contextual features and surface observation symbols, or NAVAID symbol or a surface observation symbol.
with access to METAR text), "NEXRAD" (contextual Depending on the mode of weather information selected, a
features and composite NEXRAD imagery, with access to bezel key then provides either the identifier of the airport
airport and NAVAID identifiers), and "NEX/MTR" or NAVAID, or the text associated with the surface
(contextual features, NEXRAD imagery, and surface observation symbol. Again, participants' opinions
observation symbols, with access to METAR text). The regarding the usefulness of the airport, NAVAID, and
utility and usability of the weather information displayed in METAR symbols and text are described below as
these modes is further discussed as Information Elements information elements. Participants' complaints about
in the following section. The last bezel key allows users menu navigation were primarily focused on the awkward
to turn the displayed menu labels off. When the menu method of obtaining METAR text information and
labels are on, the right most approximately 20% of the NAVAID/airport identifiers. At worst case, this required a
screen is obscured. Pressing any of the hard keys will 5-step process: selecting a scale that shows symbols of
then redisplay the displayed menu labels. Two interest (50nm or lower), selecting a weather mode
participants noted that the range scale for the NEXRAD corresponding to the information you want (NEX/MTR or
data disappears when the menus are turned off. METAR for METAR text, Graphics or NEXRAD for airport
or NAVAID identifiers), changing to crosshairs (selecting)
The joystick has three modes: Zoom, Crosshairs (for mode, orienting the crosshairs over a symbol (surface
selecting), and Scroll. The Zoom mode enables users to, observation, airport, orNAVAID), and finally selecting the
with the joystick, select one of six scales: 10nm, 25nm, bezel key to acquire the METAR text or switch between
50nm, 100nm, 200nm, and 500nm (showing the airport/NAVAID identifiers. This process was considered
continental United States). These distances refer to the onerous enough to motivate several suggestions for
redesigna,sdiscusseldater."(I)
want (more information read at normal hand-held viewing distance, and the
on those surface observations/airports/NA VAIDs that are display was difficult to read by subjects using bifocal
relevant to my route) rather than picking through (more of lenses. Participants also suggested that the aircraft
them) with (the) interface." The Scroll joystick mode symbol is distorted when not flying in a cardinal direction
allows the user to slew the viewable region of the and making it difficult to clearly see the track at a glance.
Continental US map east, west, north and south of the
currently viewed frame.
Scale Leqend - The scale legend is on the top line of the
There are three modes for the use of GPS position display, in the upper right corner, and provides a line 1.5
data: "Off," "Lock," and "Free." These modes determine cm long with end hashes followed by the number of miles
constraints on the viewable window of the available represented by this line on the display and appended with
Continental US map, and whether an aircraft symbol is "NM" for "nautical miles." Experimenters observed that
present. The GPS Free and Lock modes present an participants often used a pencil or their fingers to make a
aircraft symbol. The GPS Lock mode constrains the "ruler" for the unit of scale displayed in the legend, and
viewable region of the map to that which is centered on then determined distance from aircraft position to a
the aircraft's position, thereby providing a 3600view around display element of interest (weather, airport, NAVAID).
the aircraft for the distance indicated by the chosen scale. While no participants explicitly mentioned that they did
Therefore, in this mode, the scroll mode is inhibited, and this, their discomfort with the manner in which distance
one can only select items that are within the viewed information was conveyed is obvious in other comments.
region of the display as defined by their position relative to Four participants volunteered comments indicating that
aircraft position and the map scale chosen. Only one they would have preferred to have range rings around the
participant indicated that he had forgotten that the scroll aircraft position symbol to help determine the distance
function was inhibited during the GPS Lock mode. While between weather and the aircraft position. Another
two participants explicitly preferred the 3600 plan view participant volunteered that he would have preferred to use
around the aircraft, one mentioned that he used the GPS the crosshairs to select the display element of interest
Free mode to provide more of a forward view of the and to be provided with the bearing and distance from the
weather than backward view. Participants' perceived use GPS-derived aircraft position.
of and comments on the aircraft symbol are further
described below as an information element.
Menu Key Labels Participants' comments from
The display presents information in a North-up debriefing did not include any assessment of the menu
configuration. One participant mentioned the need for an key labels, and these were not addressed in the usability
interface control element to allow one to change to a rating scales. Observations from training suggested that
Track-up display orientation control. Two participants the term "graphics," which refers to the display mode
discussed having an interface control or automatic without weather information and only contextual and
reorienting of the display; to be North-up when used aviation symbols, was not intuitive for participants.
strategically, and Track-up when used tactically. One can Several initially misinterpreted this to mean the "graphical"
effectively convert this tethered display to a Track-up weather that was the most salient feature of this new
presentation by turning the display to orient the direction technology. Participants also noted that the menu labels
of the aircraft symbol to point up. While the text labeling weren't aligned well with the bezel buttons.
is difficult to read in this orientation, itwould provide the
desired orientation of graphical weather information, Weather Radar - NEXRAD returns were considered very
contextual features and aircraft symbol. Most participants helpful, and the colors used to encode intensity levels
did not realize this "feature" of a tethered system, as few were also considered very helpful. All ten participants
were observed to reorient the display during the flight who responded to the question, indicated that they
experiment. considered radar return color codes to represent
categorical levels of radar intensity, rather than reflecting
specific values of atmospheric phenomena (i.e., VlP
PRESENTATION & INFORMATION ELEMENTS - The levels). Five participants expressed dissatisfaction with
GWIS evaluated here provides a plan view of NEXRAD the resolution of the NEXRAD data. When using the
data, METAR symbols and text, contextual features lowest scale, units of NEXRAD information appeared too
(rivers, interstates, state boundaries, airport symbols and big to be useful. Other participants suggested appropriate
identifiers, NAVAID symbols and identifiers), and aircraft resolutions: one said it should be about 0.25nm, another
position symbol. Arrangement of information on the suggested that it should be the same as that which is
screen appeared to be fairly logical to participants. While available on onboard weather radar. Resolution was seen
two participants stated that the resolution of the display as particularly important for understanding the gradient of
(320 x 200 pixels) was adequate, three participants NEXRAD weather information intensities. Post-
desired higher resolution. Most comments suggesting the experiment debriefing sessions revealed that participants
need for improved resolution indicated that text labels were not all well informed about the construction of
(e.g., NAVAIDs, airport identifiers) were too small to be
NEXRADimages.TheconstructioonfNEXRADimages
conjunction with the finding from previous analyses from
wasnotincludeidnthetrainingoftheAWlNdisplay.
this study and others, that pilots using a GWIS are less
likely to request additional information from other sources
TheNEXRADproductcreationdateandtimeis
(i.e., Flight Watch). One participant did recognize the
displayedintheupperleftcornerofthedisplayandis
danger inherent in using delayed weather information and
shownas,fortheproducdtelivereodnJuly-_at17:41
mentioned that he compensated for this by looking only at
Zulutime(or1:41PMEDT):"NEXRAD07/0717:41Z."
weather very close to the airplane symbol. This strategy
Halfoftheparticipandtsidnotrecalul singtheageofthe
would not compensate for old weather information. At
NEXRAiDnformatioinntheiruseofthisinformatioenv,en
times during the return trip, after the experiment was over,
thougho,naveragep,articipantrsa'tingsindicatethatthe
weather information was not updated for as long as over
ageof thisinformatioins fairlyapparent.Onlyfive 40 minutes.
participanatsttestedtousingtheageinformationT.he
averageratingofhowapparenthteageinformatioisnwas
Despite this displeasure with, and failure to
about90%. Onlytwo of the six participantwsho
appreciate, the age of the NEXRAD information,
confessetdonotusingtheageoftheNEXRAdDatarated
participants rated this weather information source as fairly
thedegreetowhichthisinformatiownasapparenOt.neof
reliable. Inspection of individual scores revealed that four
theserateditabout97%apparentth,eotherabout18%
of the participants rated this information as less than
apparent.Subsequecnotmmentafroyrmsindicatetdhat
75%, where 100% is reliable, and two participants rated it
the participanwt ho confessedto not usingthis
less than 50% reliable. It is important to realize that
informatiobnu,trateditasveryapparencthidedhimself
these ratings were taken for different flights, and that the
for not usingthis and laterrememberetdhatthis
experiences during these flights may have differed. While
informatiownasavailable.Oneparticipanctommented
the lower scores for age and reliability may be less
thattheageoftheinformatiosnhouldbedetectab"leata
prevalent than more acceptable ratings, the fact that these
glance,i"mplyintghatitwasn't.Thiscommenat,longwith
reflect more challenging scenarios for the equipment
the ratingscale resultsclearlydemonstratetshe
cannot be separated from rater bias.
differencebetweena bit of informatiobneingvisually
availablea,ndbeingattendetdo,andthereforaevailabfleor When NEXRAD data were not available for a section
furtheprrocessing.
of the map, an opaque yellow bar was presented for the
width of the screen in the region for which there was no
Sevenof twelveparticipantwserelessthan50%
data. There could be several of these "missing data"
confidentthattheyknewwhatthe longestdelayin
blocks on a screen if the up-linked file was incomplete in
NEXRAwDeatheurpdatewsasduringtheirflights.Three
different places; and, where these lacking areas were
ofthesesevenparticipanrtastedtheirawarenesosfthe
adjacent, wider bars formed. Participants did not fail to
longesdtelayas0%.Sevenparticipanrtastedthedegree
appreciate the meaning of this display element. However
towhichtheageoftheNEXRADdatawasacceptable,
they question the formatting, and conditions for use.
and,onaverager,ateditlessthanfairlyacceptableI.n
Several participants expressed the opinion that at some
freeformdiscussionp,articipantcso' mmentasboutthe
point there is a tradeoff between a newer image with
ageofweatheirnformatiopnredominanrtelyferredtothat
mostly missing data, and an older image that is complete.
oftheNEXRADinformationG.eneralltyhesecomments
Two participants recommended continuing to display older
impliedthatthecurrenbtest-casuepdaterate,5minutes,
data when newer data is significantly degraded or there is
isadequatfeorstrategiucse;butthatfasterupdaterates
no service, and provide an alert to indicate that the data is
arerequiredfortacticaul se;andslowerupdaterates,
aged.
particularloyver10 minutesold, are insufficienftor
appropriatueseinflight. Severaplarticipanitnsdicated
thattheywouldappreciataenalerttoindicatewhenthis
Surface Observations - Participants viewed the surface
weatheirnformatioisn"old."Whenaskedtoassum2e00
observation METAR symbols as not particularly useful.
knotstrueairspeedf,ourparticipanitnsdicatedthatthe
Debriefing comments corroborated this rating scale result.
criteriafor this alertwouldbe at 10 minuteso, ne
While participants were instructed to consider that they
suggesteadt15minutesW. hileavionicms anufacturers
would be landing at a given destination in the given
andFAAusageguidelineisntendforpilotstousethese
scenario, three participants indicated that these symbols
GWISs only for strategicpurposes,participants'
were not very useful for the enroute portion of their flight,
commentsclearlyindicatethat theywill use these
but would be of more use if they were actually landing or
systemstosupporttacticadlecisionasswell.Thiswas
considering an alternate. METARS may also have been
particularolybviouisnresponsetosthequestion"h, owold
more useful in a weather scenario with widespread low
doyouthinktheweatheirnformatiownas?"Someofthe
ceiling and visibility conditions, rather than frontal
moreconcerninrgesponsewsere:
"1can't remember... (I) convective systems. Color-coding of the METAR symbols
assumed it was real-time;" "(I'm) so used to considering it (the upper half coded for ceiling, the lower half forvisibility)
real-time;" "(I) didn't notice.., could thread the needle with was rated as fairly helpful, and these color codes were
it;" "good enough to make a divert decision." This type of always perceived as categorical indications. While
response is particularly troubling when considered in participants generally considered color-coding to be
effectiveulysedforindividuatylpesofweatheinrformation, problemwhichincludedspokenMETARS("text
is
two participantisndicatedthatthesurfaceobservation
inexcusable now!'_, a dedicated METAR text line at the
symboclolorsweredifficulttodeterminwehencombined
bottom of the display, automatic presentation of the
withNEXRADdata.Oneoftheseparticipantass,wellas METAR text in this dedicated screen area when
a differenptarticipanmt,entionetdhattheydidn'treally
crosshairs are over a surface observation graphic, and an
useortheydidnotselecttheMETARgraphicaslymbols
automatically-generated listing of the surface observations
in orderto reducescreenclutter. Twoparticipants
that are relevant to a route - to eliminate picking through
indicatedthat they used the color-codedsurface
the menus and selecting symbols.
observationsymbolsto indicatea trendin surface
conditionosvera regiona,ndtoindicateregionswhere
conditionrsequireIFR. Oneofthesenotedthatthese
NAVAIDS/ Airports/ Contextual Features - Several
symbolcsouldrepresenotlddata.
contextual and aviation location display elements provided
users with references to understand where the weather
Nineofthetwelveparticipantrsa'tingso,naverage,
was, and how far they were along their route. The display
indicattehattheyfoundtheageofthesurfaceobservation showed state boundaries as white lines. At the 50nm
datatobefairlyapparenot,r obvious.Fourofthese
scale and below, the display showed major rivers as blue
ratingsh,owevewr,erebelow80%,aqdtwobelow50%.
lines, and interstates as yellow lines. At the 100nm scale
Thesesamenineparticipanotsn,averagefo,undtheage
and below, the display could be configured to show grey
of surfaceobservationdatato be lessthanfairly
rectangles, indicating certain airports, and blue circles,
acceptableF. iveoftheseratingswerelessthan75%
indicating certain NAVAIDS. While all twelve participants
acceptable.Commentrsegardintgheageofthedata
reported using the airport symbols, their ratings indicated
appearetdofocusontheageoftheNEXRAiDnformation,
these symbols were not particularly helpful to them. Eight
mostlikelyduetotheprioritythattheyascribedtothe
participants reported using the NAVAID symbols, and
NEXRADinformationas previouslymentioned.One
their ratings indicate that these symbols were, on
participandtidexplicitlcyommenthtathedidnotconsider
average, only marginally helpful.
theageoftheMETARinformatiownhenusingit. When
askedabouttheirconfidencineknowintghelongesdtelay
One participant complained that the airport and
forMETARinformatioeni,ghtoftheelevenrespondents'
NAVAID symbols were of limited use because those that
ratingwserelessthan52%confidenat,ndthreeofthese
are in the system database don't include those that he
expressendoconfidencientheirknowledgoefthemost
would find most useful for orienting himself along the route
agedMETARinformation.
and estimating the location of weather from aural sources.
Specifically, he questioned why TACANs, major
TheonlywaytodeterminteheageoftheMETAR VORTACs, and NDBs were not included in the database
informationwas to accessthe METARtext page.
for display. Further, the display allowed a user to display
Howevefirveofthetwelveparticipanitnsdicatetdhatthey
only one label of a NAVAID or airport at a time. Seeing
reliedprincipalloyn the graphicaslurfaceobservation
successive airports or successive NAVAID labels, only
informatiosnymbolsn,ottheMETARtext,tounderstand
requires one to re-orient the crosshairs. However, seeing
surfaceconditionsM. ETARtextinthisGWISwascoded
the label for an airport after a NAVAID (or vice versa)
andpresentefdu,llpageo,nthescreent;hereboybscuring
requires not only reorienting the crosshairs, but also
allotherinformationP.articipanctsonsideretdhecontent
selecting the appropriate menu key to change this mode.
oftheMETARtextscreenstobeveryhelpfula,ndthe
This problem was mentioned in the same breath as the
formveryeasytoread. Participandtsidnotcomplain
comments regarding the inaccessibility of the METAR
aboutcodedMETARinformatioanndallwereableto
text information, and the solution of a dedicated text line
decodeall but the specialremarksduringtraining.
was seen as appropriate for solving this problem as well.
FormaotftheMETARscreenwserefairlyeasytoread.
One participant suggested that if the screen resolution
Therewaswidevariancein the numberof METARs
were better, more labels could be concurrently available.
participanrtseporteadccessin(gRange=2,20;Mean=
Whatever the solution, the current design appears to not
8.68;Median=7.5).Onaverageth,etwelveparticipants'
adequately support users' requirements for orienting
ratingsindicatetheyconsideretdhe graphicaslurface
themselves with respect to geographical and aviation
observatioinnformatiotonbefairlyreliable.
features and the weather information displayed, "Now it
takes too much time, workload, brainpower."
Several participantscommentedthat METAR
informationshouldbe accessiblewithouthavingto
sacrificethe graphicalinformationon the display.
Airplane Symbol/GPS Modes -This display received input
SolutiontsothisproblemincludesdpokenMETARSa,nd
from a GPS to indicate aircraft position. The display has
adedicateMdETARtextlineatthebottomofthedisplay.
three modes for using this GPS information. The "GPS-
Oneparticipanetxpressetdheopinionthatthisiswhat Off" mode does not use the GPS information. The "GPS-
FSSisfor,thatitwouldbeeasierforhimtocallandhave
Lock" mode displays a magenta aircraft symbol in the
theMETARtextinformatiorneadtohimthantoreaditon
position indicated by the GPS, and locks the displayed
thissystem.Participantvsolunteeresdolutionstothis
portion of the map to be centered on this aircraft symbol.