Table Of ContentARI Contractor Report 2004-03
Perspectives On Studying
Collaboration In Distributed Networks.
Karol G. Ross
Klein Associates Inc
‘This report is published to meel legal and contractual requirements and may not
meet ARI's scientiic or professional standards for publication,
January 2004
United States Army Research Instituto for the Behavioral and Social Sclonces
Appraved for pub lease; disthuton is united
- 20040202 027
REPORT DOCUMENTATION PAGE
4 REPORT DATE Wd in| 7 RECHT TYEE ‘3. DATES COVERED tom ..t
aang 2008 int ‘Apri 2002 — August 2003,
Perspectives On Studying Collaboration n Distibuted BASWOI-02-P-0526 .
Networks 1 PROGRAM E-CVENT HUME
. 22065.
7, AUTHORS a PROT MOMBER
ara0 _.
Karol G, Rese ea WUMER
208
7 PE HACPSMING ORCAMIZATICN NANE(S) AND ADRESS) ‘3. PERFORMING ORGANIZAT ON RFPORT MUMDER.
ein Associates In.
11750 Commerce Center Bld N
arbor, OH 4524-6952
©, SPONSORINGIAONITORING ASENCY NANE:S) AND ADDRESSES, | 1 MONTOR ACROWTYT
U.S. Aimy Research Intute fre Betavral
‘and Social Suierces AR
5001 Eisemenior Avenue 1, HORTOR REPORT RUNGER
ATTN: DAPE-ARIE
‘Avera, VA 22336-4841 Contacte Repost 7008-98
Approved for publ lass dstibuien is unin,
115. suf ERTART HOTER
“Tis reports pubshed to meet legal and coriactl raqurnents and may no: meet ARs eciefe ardor
oeasiont standards fe pubicson
1 ABSTRAGT farsa Pa i
“This epor desortes cognive vanables and farnewerks thal ste useful in ha ineealgation of nctwork collaboration in
reap ervnrmant, Netwark eullaboralin isc tent Under shady by the U.S. Army Research nse forthe
‘Behavioral and Social Sciences (Akl}ir he covert of performance in a sinulad network task using "he game
SCUDHunt. SCUDHunt was develoned by ThoughiUnk, Ioarparate: ‘er to Dofonse Advanced Research Projects
‘Agency (DARPA), Thw use of SCUDHunt does net conetivie erdorsomntofthe product by ARI fhe U-S. Army or tha
US Department ot Detenee. Kay verables tat fcifae funtion ht silted network enironment sre
tiscussed. based on ie resus of inlervews wth profeiont SCUCHUnK players. Frameworks fr examin ng the dats
frum te SCUDHntctsies are presertod, Finaly. bl reccrsmendtions for he dracions of furs research ints
domain ae presente.
Copritve Task Analysis Callateraton, Communication Glassiesticn, Distibsted Tearswork, Net-Centic, Network
Contin, SCUDHunt, Stualinal awareness
= cin engerTy Ses
pasteer oe none one
Urstimted Geriasesaes
Unclassified
4a, TaSPAGE
‘Unciaesteg
| 97, anstracr
UUrdlassed
Klein Associates te Commer DAS WOLD P0596
Purpose
“The parposs of this report is to deserihe cognitive variables and frameworks that
ate uceful inthe investigation of network co¥aboration in Army cavironments, Network
collaboration is currently under stusly by the U.S. Army Research Insitute inthe content
‘of performance imu simulated network task using the SCUDHunt” gure, developed by
DARPA, as purt of a study of distributed eltutional awareness (Perla, Markowitz, Nofi,
‘Weuve, Loughtan, ee Stahl, 2000), Key variables that Jaclitate metioning inthis
avironment will be discusced, base on Ihe results oF iocerviews with proficicut
SCUDInat players, Frameworks for examining the data from the SCUDHunt studios are
also presented, Finally, bricf recommendations for the directions of fnture research in this
domain avo prescntod
“These perspectives on collaboration rescatch aro documented in response lo the
cod to Heller understand the emerging notworked organivationa inthe Army’ (Aber,
Garstka, & Stein, 1999), In ths batiloficlds where today's soldiers apply their skills,
‘commrunication is both more complex and more importanc than cver before, Military
perations are widely distrbuted over space and time, and this necessitates x similar
distribution for the individuals involved in those operations. There ean he no arguement
the collaboration al shared understanding of tho batlespace a required for effective
functioning between units fa this situation, bar ths individual sills (hat upport this
‘process ane aot well understood, What skills influence tho successor failure of
‘ommunication benvecn soldicrs? Whon these skill ane identiGed, wat steps must be
tac in order to facilitate individuals ia become team mrembers who are fluent in the
cffoetive methods of netoork collaboration? These skills are crucial for the U.S. Army’
fovees co maintain thelr curren level of information and technical superiority
‘Commnnication may tke place on the ballefield under a variety of
circumstances, and with varying goals. ising uetwork resourocs for collaboration
purposes has become a panicularly atvactive option, becanse of their versallity wd the
possibility for eapid delivery of txt and graphic file, such as fragmentary orders and
tactical maps, There are als polenta plfuls associated with a nerwork-based mncthod of
‘communication, hough. Many soldiers may be unfamiliar with the technology, or he
technology may be non-intuitive and difficult to use: the necossary hardsware and
software may nat bo cerizely voli cme pressure muy preclurle the use of certain
_neteds of nctwork communication, beceuse of delays in tansfer. But, most important
solders may just nat be skilled in how to collaborate inthis ype of envionment, This is
‘a bigh-stakes envizonment andthe skill to use the emerging technolagies must include
‘highly haned collaboration sills, Resoarch to understand the varies in network
collaboration is erical to enpport the development uf training for collaboration skills,
yh we of ACUI Tunl doe nt conse endreeet ofthe producthy ABI the CLS. Army oe UTS,
Depoctneat of Defense.
‘lein Acacias oe. Contoet ASTWOI-D2-P.0526
“Understanding Collaboration
Cognitive Task Analysis
‘One method for exploring the skills ‘hat will be nocded in notwork-collaborative
cenvitomments is Cognitive Task Analysis (CTA), CTA is act of techniques (eg, Klein,
Calderwood, & MucGregor, 1989) that were designed to ake advanage of the exporicuce
‘of a subjoct-malter expert (SME) in order to understand & complex provess, job, oF task,
\sposially in euations where tino prossure, high stakes, and ambiguity of information
are preseal, CTA allows the roscarchcr to probe the experiences of the expert for insights
iano exaenly how exitical decisions were mutle To the context of network collaborations,
experts wha have sed the relevant technology many times, under the environmental
‘conditions of interes, can provide valusble information sbout what makes this process
<ifficull and where attention must bo fosusce in order to improve performance, all lout a
ers point of view.
Ibis difficult o fina such expense as these euvizouments are only naw emerging,
in responso te changing onissions and technologics. To envision thin mew allied, the
SCUDFunt gure is being used to simulate that environment. To understand the
‘eallahoration skills that may be needed in the mew hattleffeld environment. Klsin
Astociatca condusted a CTA using researchers who were proficicnt at collaborating in
the SCUDHunt context. The purpose of the CLA was to suggest the Qpes of
collaboration skills tha ahonld be considered wher. the SCUDHuut data ae analyzed,
Task Diagram
‘A Task Diagram (Milivllo, Hutlon, & Miler, 1996; Miltello, Hutton, Piske,
‘Knight, & Klcin, 1997) was used Mest fo lay out the major tasks inthe simulated
‘environment andl lo identify which tasks are most cognitively challenying, Three
experienced SCUDE unt playese were intervicwod to determine ho primary tasks in
playing the game, The interviewees were asked to come up with three to six mais tasks
that caver the performance inthe gume and then 1 indicate which of these tasks were the
msl cognitively challenging. Later, the results of the Task Diagram were compiled into
‘one graphic representation reflecting the input ftom all xc inlerviews, arul via Task
‘Diagram was reviewed with the intorsiewces before Finalization
‘The finol Task Diagram synthesizing the interviews with thros proficient
‘SCUDHnt players is shown in Figure 1. Twelve tasks were identified as comprising
SCUDHunt performance:
‘© Know own and partner's assets
‘© Genecate own plan for asset placement
‘© Coordinate own plan with parmer's plan
‘tein Asoc f, Contact DASWOLIDP-0526
+ Deploy own assets
+ Remember where assts ure deployed
+ Look a! share cesalis of ineligenoe gathoring
+» Coordinate ta fiud out which assols were used by the partner to understand
results
Consider reliability of assets to assess resulls
+ Consider any provious intelligence and interpretations
Generate own Strike Plan
Trpot Surike Plan and get rors
© Generate now asssl placement plan to refine conclusions.
‘mere yseL UNHCR
2g sssmof sees ee
pena PORES OT
Scsreti-romEvavennn03 5 SIEOREY OI
‘loin Asoc le, Contact DASWOI-O2 P0526
Of the 12 SCTDHunt tasks, 5 woro identified by the interviewees as cognitively
challenging. These tasks include the following:
+ Generate own plan for asset placement
‘Coordinate own plan with parte’ plan
4 Consider relshility o asets to assess osu
+ Consider any previous intelligence and interpretations
«Generate own Strike Plan
Knowledge Audie
A Knowledge Audit (Klein & Militell, in press) was used to further assess the
‘ive casks that were identifisd as cognitively challenging to understand the nature of the
challenges and the ealluboralive aspects of the tasks. The Knowledge Andit consists of 8
set of probes designcd to extract expertiae along a aumber of dimexsions: widerslanding
the big picture, noticing irmportant cues, strategies for working smart, noticing
‘opportunites for occas, und selC-monitoring. The interview is based on obtaining
‘ckamples of euch of these dimensions and probing to understand the cue and stotegies
the proficient or expert performer user, as well as what they believe makes the task
difficult for tho nove.
‘The results of the thee interviews were compiled inia ene Knowledge Audit ble
and the strategies, cnos, and difficultice were reviewed to see What cognitive skills and
‘fmetions were indicated basell on the Trameorks reviewed balow. The following tasks
‘were combined io the final able of interviews esults, because the findings were 50
intertwined: consider reliabizity of assets co assoss Tosults, consider previous intelligence,
and generate own Seike Plan. The findings are displayed in Appendix A.
Coynitive Frameworks for Understanding Collaboration Rehaviors
Tince sources were used as potentiat categorizations of cognitive functions. These
Frameworks are showa in Appendix B. The first source is a report fiom x cognitive
tesfing project that was conducted for the Navy (O'Donnell, 2001). A eognitive model
and final list of cognitive functions for the project are shown. The second is the list of
‘taining cogmitive abjectiven that see generated for team decision skits training for
Firefighters (Gurrs, Melek, Ross, & Thordscu, 2002), ‘Tho third is a matrix of counitive
skills generated as part of a situation awarauers project (Klein, 2001)
‘Whion the results of the CTA ani the cognitive (ramewrorks wore combined, a ist
‘of taks, kills, apa functions was generated, These ate shown in ‘Ibe 1 below.
ein Asa ne ‘Contet PASI 02-7-0526,
Table 1
Cognitive Elements in SCUDFlunt Performance
Cognitive
‘© Understand own capabilities (systems and personnel)
© Envision goals and plans
‘© Define roles aad functions
‘© Unsestand enemy capabilities and behaviors
Assess the situation
‘© Balance irarnediate concerns and longer term objertives
= Corpensate to fil gaps in tea porformance
Cognitive Skills
‘© Managing altention
1s Diseeiminsring eves and pattems
| © Constructing mental models of cause wud e(Tect (inerpret infomation)
‘© Building 2 kloey frora incomplete infomation
‘© Decentering (understand hov own actions affoet partner)
Monitoring one’s own perfornamee (e.g, avoiding fixation, avoiding over-
enphasis on most recent information, realizing witen expectancies have heen
violate)
Cognitive Panetions
‘© Pereoplual processing
+ Short-term memory (o maintain overal picture of search arey ao td
procedural and doolarative knowledas)*
+ Wosking memory
* Tia this case, declacalive and procedural nowledge are not retrieved from Tong
term memory for usc in working memory, but aro all eld in shoe term memory while
other skiflsifmctions are excoulad, Memory aids (cheat sheets) ate used 10 suppatt short-
term memcry.
lin Arnsates as, (Contact DASWDL 02-P-0526,
Communication ant Distributed Teamwork Kramoworks
“for Understanding Collaboration
Teams Communication Variables
A communication framework was suggested as a manner of investigating,
collaboration behaviors evidenced in SCUDHut. The framework war eryanived 10
facili « comtect analysis of te SCLDHmt data, Conlect anatysis "a research
lechnigue for making fnfereuecs by systematically and objectively identifying specified
characteristics within a text” (Stone, Dunphy, Smith, & Ogilvie, 1966). For those
inuerested in exploring couunuication, con‘en| analysis allows for the examination and
‘deatifcation of messages found ina text, Content analysis was a very popular method of |
analyzing public and mass medinled vornenunication in the 1970s and 1980s when
hundreds of jocmal articles an television programming and political communication were
published using thie meduedology (Comsicck, 1975; Jaskson-Feevk & Kras, 1980).
‘fhe main goal of comlent analysis is o describe the chatacterstes of messages
camber in texts. Krippsndorf (1980) identified four advantages to content analysis.
Fim, itis unobirusive tecanscitstudics texts that alcady oxist, Second. it wocepts
unstructured material. which obscrvers eategorive, There is no need for simmetured
interviews or quostionnaies, Third, content analysis allows a researcher to emdy dats
‘within a context. Last, iti able to handle massive amounts of data,
Ono type of content analysis it convenation enalysis. Conversation analysis,
cxarines mesnges exchanged daring dyadic or small group interactions in onder to
scaver the “systematic and orderly propctics which are maningfol to conversents [al
escatehcrs|" (leritage, 1989), Ths basie process af enversational anelysis is to obtain
conversation da, transeribe From oral to writen form, develop coding schemes,
calegrize messages inta schemes, analyze codings ta dscribe and draw inferences about
the content, stmctur, or effects of conversation, and report th: findings.
It should be notod tha! suzgestions and crilicians have heen made about the
rolisbility and objectivity of conten! analysis. Spetiiealy. Kofbe and Burnet (1991)
suggest that objectivity af eantent analysis studios can be impzoved by precise definitions
and operational rules and by utilizing codars that are indapeudent of both the authors of
the research and each other, They go on to suggest thal tho metboxt for determining
inkereode reliability should be documented in the reseacch
A framework for performing a content anelysis of the SCUDHun! data is shown
in Appendix C. Itinoludes variables aseocialed with “Task Roles,” “Tear Maintenance
Roles,” and “Self-Oriented Behaviots.” In addition ta Appendix C providi
framework, it also includes part of the SCLDHunt data coded into the appropriate
categories ae a sample of tow the analysis could be earried out
‘There are generally rvo types of roles individuals nood to teke-on fo make s team
function in collaborative mennst—Ihove concerned with completing Ue fears tach
sein hsociates Te. Contact DASWDI-UZ-P-SI6
nal those coneemed withthe social and cmotionst needs of the group (Ketrow. 1991).
‘Bath types af roles are cescntal ta an effective team. Task roles are concerned with
getting te jab done. Those who function in task roles help move the grou fowant ita
‘goal and help lhe group be aluptable to needs and changes along the way. Micnfenance
roles are convernel wilt he team getting along, and facus on developing, ways la express
and del with eraolianal dimensions of working ina group, In addition, there are self
ssentered roles that are identified as ween individuals focus on their oven n2eds, which
Alsirats from collaharation. Mast effective team members will he able to adapt to tace on
‘a viriely of these roles as needed (and avoid the selE-centered hehaviors} to keep the tear
functioning effectively, Ihe categories forthe coding scheme are derived from Mare
(1994) and Keto (1999),
‘Ta catty ont the content analysis, 4 minimum of fur cadens should be used,
‘There should be an equal division of men #nd women t control for gender differences in
perception of bohaviors, Each should be trained in the coding procedures, They should be
‘ven a coding shect along with an exploration of the procedures. They should be
‘qucstioned to determine their underatanding of these procedures, ‘ogether, cars shoul
‘gonduct u mack analysis af oue interaction, Then coders should cade a second interaction
separately, After complcting this interaction, coders shold meet ta discuss their degroc
of agrooment, Afler Ihe initial waining, coders should be assigned to code ivaraetions
nepurately. OT uvese intezaetions, two should overlap to sorve asa measure of iniercoder
reiabity
Disivibuied Tearovorke
From our work with distributed teams (Wiggins & Klinger, 2001) we compiled a
umber of challenges or points at which distributed tearmvorc cun break dawn, These are
cxeliages of information, yal eonllicis, synchnnizaiion averhead, diagnostic overhead.
builling wal rnaintsining situation awareness, dfstributingsitustioual awaronoss, and
sigidity. We used these clomenss to construct anether matrix that could be used fo analyze
the collaborative exchanges during SCUIDHluet. This matris is shown in Appendix D.
‘The matry also provides an exaniple of sonia of the SCUDHunt dita coded into there
categories.
Future Research
-Funue research rmust examine the bchaviors of trained persennel eollaborating in
sich contexts, instances of the distributed teamwork breslalown discussed ahave must he
caplured and analyze to understand the ups in performance. This very likely that for
lower-level antdirs ta be good collaborators, thoy will require training that quickly
sttengthus their tactical thinking skills even at lower ranks and eshctons so they eon
recognize what is important to report. We must also increuse their awareness of hey areas
in which distributed (sarmork ean break down, and tein behaviors to monitor and repair
such breakdowns. This may include taining in task roles, team maintenance roles, mid
self-monitoring rotes indicated inthe research frameworks discussed here,
