DTIC ADA525552: Integrating Methodologists into Teams of Substantive Experts

10 Pages·0.57 MB·English

by  Defense Technical Information Center

Reducing Analytic Error
Integrating Methodologists into Teams of
Substantive Experts
Rob Johnston
Intelligence analysis, like other energy necessary to generate new
complex tasks, demands consider knowledge in that field. It takes a
able expertise. It requires considerable amount of time and
individuals who can recognize pat regular exposure to a large num
terns in large data sets, solve ber of cases to become an expert.
complex problems, and make pre
dictions about future behavior or An individual enters a field ofstudy
events. To perform these tasks suc as a novice. The novice needs to
(cid:147)
cessfully, analysts must dedicate a learn the guiding principles and
considerable number of years to rules(cid:151)the heuristics and con
Paradoxically, it is the researching specific topics, pro straints(cid:151)of a given task in order
cesses, and geographic regions. to perform that task. Concurrently.
specificity of expertise
the novice needs to be exposed to
that makes expert Paradoxically, it is the specificity of specific cases, or instances, that test
forecasts unreliable. expertise that makes expert fore the boundaries of such heuristics.
casts unreliable. While experts Generally, a novice will find a men
outperform novices and machines tor to guide her through the
in pattern recognition and problem process of acquiring new knowl
solving, expert predictions of future edge. A fairly simple example
behavior or events are seldom as would be someone learning to play
accurate as simple actuarial tables. chess. The novice chess player
In part, this is due to cognitive seeks a mentor to teach her the
biases and processing-time con object of the game, the number of
straints. In part, it is due to the spaces, the names ofthe pieces, the
nature of expertise itself and the function of each piece, how each
process by which one becomes an piece is moved, and the necessary
expert.1 conditions for winning or losing the
game.
Becoming an Expert In time, and with much practice,
the novice begins to recognize pat
Expertise is commitment coupled terns of behavior within cases and,
with creativity. Specifically, it is the thus, becomes a journeyman. With
commitment of time, energy, and more practice and exposure to
resources to a relatively narrow increasingly complex cases, the
field of study and the creative journeyman finds patterns not only
within cases but also between
cases. More importantly, the jour
More than 200 individuals contributed to neyman learns that these patterns
this study The author is indebted to the re often repeat themselves over time.
Dr. RobJohnston is a stheearSchteurdsy.offelilnotwesl,liagenndces,tatfhfeatIntshteitcuteentfeorr Dfocr- The journeyman still maintains reg
pthoestCdIocAtoCreanlterresfeoarrtchhefSetlulodwy aotf gttneecnn.s,ceeaAnAndsaslAoycNsiSeasEt,iRotnih,necENvaiSttdiaeofnfncaaelndBMiaslstietudadreyRnetissnetaaetrlcltihh,e supleacricfiocntparcotbwlietmhsaamnednlteoarrntomosroleve
CIA University, theJoint Military intelligence complex strategies. Returning to
Intelligence and a member of the college, the Naval Postgraduate school, Co the example of the chess player,
research staff at the Institute for alnumdbiYaaleUniUvneirvseirtsyi,tyGealosrogceotnotwrnibuUntievdertsoittyh.e the individual begins to learn pat
Defense Analyses. project. terns of opening moves, offensive
57
Report Documentation Page Form Approved
OMB No. 0704-0188
Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and
maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information,
including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington
VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it

does not display a currently valid OMB control number.
1. REPORT DATE 3. DATES COVERED
2003 2. REPORT TYPE 00-00-2003 to 00-00-2003
4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER
Integrating Methodologists into Teams of Substantive Experts
5b. GRANT NUMBER
5c. PROGRAM ELEMENT NUMBER
6. AUTHOR(S) 5d. PROJECT NUMBER
5e. TASK NUMBER
5f. WORK UNIT NUMBER
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION
Center for the Study of Intelligence,Central Intelligence REPORT NUMBER
Agency,Washington,DC,20505
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S)
11. SPONSOR/MONITOR’S REPORT
NUMBER(S)
12. DISTRIBUTION/AVAILABILITY STATEMENT
Approved for public release; distribution unlimited
13. SUPPLEMENTARY NOTES
Studies in Intelligence, Volume 47, No. 1, 2003
14. ABSTRACT

15. SUBJECT TERMS
16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF 18. NUMBER 19a. NAME OF
ABSTRACT OF PAGES RESPONSIBLE PERSON
a. REPORT b. ABSTRACT c. THIS PAGE Same as 9
unclassified unclassified unclassified Report (SAR)
Standard Form 298 (Rev. 8-98)
Prescribed by ANSI Std Z39-18
Expert Teams
Apprenticeship. is stifi
a standard method of
training for many
and defensive game-playing strate complex tasks. longer it takes to build expertise,
gies, and patterns of victory and or, more accurately, the longer it
defeat. takes to experience and store a
large number of cases or patterns
When a journeyman starts to make
and test hypotheses about future
behavior based on past expefi The Power ofExpertise
ences, she begins the next
transition Once she creatively gen instances, however, there are tens, Experts are individuals with special
erates knowledge. rather than if not hundreds, of thousands of ized knowledge suited to perform
simply matching superficial pat potential patterns. A research study the specific tasks for which they are
terns, she hecomes an expert At discovered that chess masters had trained. hut that expertise does not
this point, she is confident in her spent between 10.000 and 20,000 necessanly transfer to other
knowledge and no longer needs a hours, or more than ten years, domains. A master chess player
mentor as a guide(cid:151)she becomes studying and playing chess. On cannot apply chess expertise in a
responsible for her own knowl average, a chess master stores. game of poker(cid:151)although both
edge. In the chess example. once a 50,000 different chess patterns in chess and poker are games. a chess
journeyman begins competing long-term memory.2 master who has never played poker
against experts, makes predictions is a nuvice poker player. Similarly,
based on patterns, and tests those Similarly, a diagnostic radiologist a biochemist is not qualified to per
predictions against actual behavior, spends eight years in full time med form netirostirgery, even though
she is generating new knowledge ical training(cid:151)four years of medical both biocheniists and neurosur(cid:151)
and a deeper understanding of the school and four years of resi geons study human physiology- In
game. She is creating her own dency(cid:151)before she is qualified to other words, the more complex a
cases rather than relying on thc take a national board exam and task is, the more specialized and
cases of others. begin independent practice.3 exclusive is the knowledge
According to a 1988 study, the recluired to perform that task.
The chess example is a i(cid:146)ather short average diagnostic radiology resi
description of an apprenticeship dent sees forty cases per day, oi(cid:146) An expert perceives meaningful
model. Apprenticeship may seem around 12,000 cases per year.5 At patterns in her domain better than
like a restrictive 18th century mode the end of a residency, a diagnos non-experts. Where a novice pci(cid:146)
of education, but it is still a stan tic radiologist has stored, on ceives random or disconnected data
dard method of training for many average, 48,000 cases in long turin points, an expert connects regular
complex tasks. Academic doctoral memory. patterns within and between cases
programs are based on an appren This ability to identify patterns is
ticeship model, as are fields like Psychologists and cognitive scien not an innate percepttial skill;
law, music, engineering, and medi tists agree that the time it takes to rather it reflects the organization of
cine. Graduate students enter fields become an expert depends on the knowledge after exposure to and
of study, find mentors, and begin complexity of the task and thu experience with thousands of
the long process of becoming inde number of cases, or patterns, to cases.6
pendent experts and generating which an individual is exposed.
new knowledge in their respective The more complex the task, the
domains. Ni Niinskv and S Papen A,1,jiczaIIn/c/li(cid:151)
genoa(Eugene, OR Oregon Staie System oi
Higher Education, 1974). I \(cid:145)nss and 1(cid:146), Pnsi.
To some, playing chess may appear 2W chase and H Simon, (cid:147)Perception in (cid:147)On ihe 5,,Jving ofill-Siruc-inred Prohienis,(cid:148)
rather trivial when compared, for cphpess5,5(cid:146)-81Cogiiilii(cid:146)e Psychnlngi(cid:146). \roi 4, 1973 NiC)clAiik,inR. G1linais/ce(cid:146)tr.c,raj(cid:146)~n-idrcMIi(cid:145)tFealr,r,,readtsKu. Onpo(cid:145)IcCc/dge
example, with making medical American College ufRadiology Personal (london Pinn, 19801. D Egan and B
diagnoses, but both are highly com cnmmt.inicaiion, 2002 Schwaaz (cid:147)cirinking in Recall ofSymbolic
plex tasks. Chess has a well- GlAaseLre,sgDoidK,lopHferR,uhainndsoYn,WaPngF,eli(cid:147)Eoxvpiecrht.isKe in Di9r7a9w,inpgps,(cid:148)i-ii9/-ain5i8o.n(cid:146)KanNidcKCeoigi;l,iein/,iouJ, Rveoliim7a,n,
defined set of heuristics, whereas a Compiex SkiH Diagnosing x-Ray Pictures,(cid:146) H Rueier. and S Hirile. (cid:147)Knowledgc Organi
meenddiecdalanddiavganroisaeblse.seeInmbmootrhe open NlliuhraeuClnio~if,AFsRrstpoGecriiraaistseeers,.(Haiinl9ld8s8dN)aiie.FanN(cid:146),J edLsaw,rTenhceeME,r zg1ar9ta8im1u.inneparpnsd,(cid:148)30SkC7io.lg3lu2uD5,itfnfe(cid:145)erPesnic(cid:146)ce(cid:146)hosloigni(cid:146),Comvpoulteir~,Pro(cid:151)
58
ExpertTeams
(cid:147)
An actuarial table is as
good, better, than
or an
expert at making calls
Experts have a deeper understand about the future. ability of experts to make fore
ing of their domains than novices casts (cid:176) The performance of
do, and utilize higher-order princi experts has been tested against
ples to solve problems.7 A novice, actuarial tables to determine if the)(cid:146)
for example, might group objects are better at making predictions
together by color or size, whereas than simple statistical models. Sev
an expert would group the same enty veai(cid:146)s later, with moi(cid:146)e than
objects according to their function two hundred experiments in differ
omarbelauetnsiiliwntiyg.thofEdxidpfaefterartesanntcdocrmwipteerireigahhewvniatdrhiitnhe Biicenetsstt,earnecxaeptserswethlsfe-arrmeeontmihtoeoryreihnaagwvtaehracenoomnfmoivt eaenqntusawdleormaamiisonunsno,t.6itofisIfdcasltueaparpalbtihoeaudtttwahietpharan
tEhxepierrtdsomraeicnosgnibeztetevrartihaabnlensovtihactes. tperdobelrermo.r1s2orEfxaipleerdtstochuencdkertshteainrd a tgiocoudla.rocrasbee,ttaern,atchtauanriaanl etxapbleertisatas
have the largest influence on a par solutions more often than novices making calIs about the futu c.
atitctuelnatriopnroobnletmhosaendvalroiacbulsest.heir ainngd irnefcoorgmnaitzieonwhneecnestshaeryyafroer miss sEpveecnifiifcacnaseexpienrftorimsagtiivoenntmhoanreis
available to the statistical model,
solving a problem(cid:146)3 Experts are
Experts have better domain-spe aware of the limits of their domain tfhoermextpheertacdtuoaersianlottabtleen.d(cid:145)7 to outper
knowledge and apply their
cific short-term and long-term
domain(cid:146)s heuristics to solve prob
moveemr,orexypetrhtasnpneorvfiocresm dtoas.k(cid:148) s Minortheeir lexepmesritheantcefalblaosuetside of their Trehseeraercahrefifnedwingesx,cebputtiotnhse teoxctheepse
domains faster than novices and
c1commmsiotlvifnegw.e9r eInrtreorresstwihnigllye,perxopebrts The Paradox ofExpertise etixopnesrtasrearienfgoirvmeantitvhee. reWshuletns of the
go about solving problems differ
ently than novices. Experts spend (cid:145)l(cid:146)he strengths of expertise can also (cid:145)~ FL Reichenhach, EsperienccandPrethctitu,
more time thinking about a prob he weaknesses)(cid:146) Although one (cid:145)chicago, IL University ofChicago i(cid:146)ress,
lem to fully understand it at the would expect experts to he good 1938), 1 Sarbin, (cid:147)A ConirilautiontoiheSisidy
ofAciuarialand individual Med odsofPreclic(cid:151)
beginning of a task than do nov forecasters, they are not particu lion,(cid:148) AniencanJoz,rnalofSoc~ologs(cid:146), Vol 18,
ices, who immediately seek to find larly good at making predictions 1943, pp 593-602
a solution.(cid:146)0 Experts use their about the future. Since the 1930s, ViSergsuDsaAwcetso.ai0ialiJ(cid:146)auusdig,meanntd,(cid:148)aScMieee,,hcl,,,V(cid:147)colli.ni2c4a3l.
knowledge of previous cases as researchers have been testing the i989, pp~ i668-i674; W Grove and i(cid:146), Mcclii,
context for creating mental models !Comparanve Efficiency ofinformal IStibiec
live, impressionistic) and Formal (Mechani
to solve given problems. cal, Algorithmic) Piediction Procedures(cid:146) The
v0,~5 and T Posi, (cid:147)On the Solving ofill clinical-Statistical Coniro~(cid:146)ersy,(cid:148) Psicholog).,
Struciured Problems(cid:148) Ni Chi, R Glaser, and Pu/thePolw~(cid:146). andan(cid:146), \(cid:145)ol, 2, No 2, 1996,
Ni Farr. eds, Op Cil pp 293-323
NI Chi, P Peliovich, and R Glaser, !caiego iaNi Chi. H Glaser, and E Rees, Expertisein i7 R Dawes, (cid:147)A Case Study ofGraduate Ad
nzat,on and Represeniaiion ofPhysics Prob ProblemSolving.(cid:148) R Siernberg.ed Advances missions Applicaiion ofThree Principles of
lems by Experts and Novices,(cid:148) Cognitive ui/hePs)(cid:146)cholopj(cid:146) nj/lumen Intell.igence Human Decision Making, A,nencanJ(cid:146)sychol
Science. VoL 5, i98i, pp~ i21-125. NI, We,ser (H,llsdale. NJ Lawrence Erlbauni Associ ogist, voL 26, i971, pp. 180-188; \V, Grove
and J sheria. Programming Problem Repre aies.i982); 0 Simon and H Simon, (cid:145)individ and P NIeehl, Op. Co (see fooinoie 16), H
sentation in Novice and Expert Program ualDifferences inSolving PhysicsProblems,(cid:148) Sacks, (cid:147)Promises, Performance, and Princi
mers.(cid:146) lnctrucltoi;al/onrnalo,filfasi(cid:151)iiIach,ne R Siegler. ed,, Children~Thinking, WhatDc ples An empirical said)(cid:146) ofParole Decision-
Studies, vol 14, 1983, pp 39i-396 ir/ops(cid:146) (Hillsdale, NJ(cid:146) Lawrence Erlhaum As making in Connecticut,(cid:148) Co,z,,ect,cutfan,Re
W chase and K Ericsson, (cid:147)Skill and Work sociates, 1978) v/en(cid:146), Vol 9. 1977, pp 349-422, 1 Sarhin, (cid:147)A
ing Memory,(cid:148) G Bower, ed ThePsi(cid:146)chologr 3 Larkin, The Role ofProblem Represenia Conirihution io the Siudy ofActuarial and in
oj(cid:146)Leanii;ig a,,dA!oti,(cid:146)atio;, (, NewYork, NY t,on in Physics,(cid:148) 0 Gentner and A Sievens, dividual Methods ofPrediction,(cid:148) A,oencan
Academic Press. 1982). eds Men/a/Modelsti-tillsdale, NJ La(cid:146)~(cid:146)rence JoiirnalofSoctologi(cid:146), 1943, pp 18, 593-602,J
ow, Chase. (cid:147)Spatial Represcniaiions ofTaxi Erlb,aum Associates, 1983). Sawyer, (cid:147)Measurement and Piedicuon, Clini
Drivers.(cid:148) 0. Rogers andJ, Slohada. eds.. Ac ii C, Camerer and E Johnson, (cid:147)The Process- cal and Statistical,(cid:148) Pc)(cid:145)chologica/ Riillelin,
qziisitioo oJSs(cid:146)nibohc Skills (New York, NY(cid:146) Performance Paradox in Expert Judgment Vol 66, 1966, pp 178-200,W, SchofieldandJ
Plenum. 1983). Ho(cid:148) Can Experts KnowsoMuchandPredict Garrard. Longiiudinal Study ofMedical Sir,
~5J PaigeandH, Simon, (cid:147)cognition i(cid:146)rocesses so Badly(cid:148) K Ericsson and J Smith, eds, To dents Selected forAdmission io Medical
in SolvingAlgebra \Vord Problems,(cid:146) B Klein ,,(cid:145)ard aGeneralTheoryofK~penise Prospects SchoolbyAciuarialandCom,nitieeMethods,(cid:146)
muniz, ed Prohlr(cid:146)ni Solving(NewYork, NY nullLuoit(cid:145)cambridge. UK Cambridge Un, 13r1,shJournaloJ(cid:146)ilethcal Education, Vol 9,
Wiley, 1966,(cid:146) vcrsiiy Press, 199i) 1975. pp 86-90
59
Expert Teams
(cid:147)
Forecasting is
complex,
a
interdisciplinary,
actuarial predictions, for example, dynamic, and (cid:149) Pattern bias(cid:151)looking for cvi
they tend to score as well as the dence that confirms rather than
statistical model if they use the sta multivariate task. rejects a hypothesis and inadvert
tistical information in making their ently filling in missing data with
own predictions.(cid:145)~ In addition, if data from prevkuis experiences.
an expert has privileged informa
tion that is not reflected in the
(cid:149) Heuristic bias(cid:151)using inappropri
statistical table, she will actually
perform hetter than the table. A Theorists and researchers differ aptreedgiuctiidoenlsi.nes or rules to make
classic example is the broken leg when trying to explain why experts
argument: Judge X has gone to the are less accurate forecasters than
theater every Friday night for the statistical models. Some have The very method by which one
past ten years. Based on an actuar argued that experts. like all becomes an expert explains why
ial table, one would predict, with humans, are inconsistent when experts are much better at describ
some certainty, that the ludge using mental models to make pre ing, explaining, performing tasks.
would go to the theater this Friday dictions. That is. the model an and problem(cid:151)solving ~vithin their
night. An expert knows, however, expert uses for predicting X in one domains than are novices, but, with
that the judge broke her leg Thurs month is different from the model a few exceptions, are worse at fore
day afternoon and is currently in used for predicting X in a follow casting than actuarial tables based
the hospital until Saturday. Know ing month, although precisely the on historical, statistical models
ing this key variable allows the same case and same data set are
expert to predict that the judge will used in both instances. ~ A given domain has specific heui(cid:146)is
not attend the theater this Friday tics for performing tasks~~~incI
night. A number of researchers point to solving problems. These rules are
human biases to explain unreliable a large part of what makes up
Although this argument makes expert predictions During the last expertise. In addition, experts
sense, it is misleading. Forecasting 30 years, researchers have catego need to acquire and store tens of
is not simply a linear logical argu rized, experimented, and theorized thousands of cases within their
ment hut rather a complex, about the cognitive aspects of fore domains in order to recognize pat
interdisciplinary, dynamic, and mul casting.2(cid:176) Despite such efforts, the terns, generate and test hypotheses,
and contribute to the collective
tivariate task. Cases are rare where literature shows little consensus knowledge within their fields. In
woeniegkheeydvaaprpiraobplreiaisteklnyotwondeatnedr rtieognasrdoifnghutmheancabuisaess oNronmeatnhiefleesstsa. other words, becoming an expert
mine an outcome. Generally, no there is general agreement that two yreeaqrusireosf vaiseiwginnifgictahnetwnourmlbdetrhroofugh
single static variable predicts types of bias exist: the lens of one specific domain. It
behavior; rather, many dynamic is the specificity that gives the
variables interact, weight and value
expert the power to recognize pat
change, and other variables are J~ Fries, c(cid:146)t at, (cid:147)Assessmeni ofRadiologic terns, perform tasks, and solve
introduced or omitted to determine Pdorooigirzeesds,iocnonitnrRolhleeudmaTntaoli.d(cid:148) AA,rtlhhrnittiissRhAeu,Rnain problems.
outcome. as written by author vol 29, No i, i98fi, pp
1-9
2e4)sjanEdvanCso,nsBeiqauseinnceHsiu(nHao,v,e,ReaU.KconiLna8wreCnacues Paradoxically, it is this same speci
L Goldberg, (cid:145)simple Models orsimple Pro ErlbaumAssociates, i989), R Iteuer, Pci(cid:146)chot ficity that is restrictive, narrowly
cesses? some Research on clinicalJudg ogyoJlnteltigeiuceAuualis/s(Washington, DC focusing the experts attention on
merns,(cid:148) AinencanPsi(cid:146)cbo/ogist. VoL 23, i968, cenierfortheStudyofintelligence. i999), D
pp 483-496, L Goldberg, (cid:147)Man versusModel Kahneman, P Slovic. and A. Tversky,Ju,d8- one domain to the exclusion of
ofMan A Rationale, Plus some Evidence, for uncut UnderUncertain/i lieunsticsandBi others. It should come as little sur
a Method ofimproving on clinical tnfereoc ases cambridge, UK(cid:146) cambridge University prise, then. that an expert would
Cs,(cid:148) Pssr.holag,ealBulletin, Vol 73, 1970. pp Press. 1982): A Tversky and D Kahnemao,
422-432, ft Leli and 5. Filskov, (cid:147)clinical-Ac (cid:147)The Beliefin the Law ofsmall Numbers,(cid:146)(cid:146) have difficulty identifying and
tuarial Detection ofand Description ofBrain PsychologicalBulletin, Vol 76, i97i, pp i05- weighing variables in an interdisci
it,m(cid:148)Jpoauirrnmae/ntowfiCtlh/utih/ecaWlePcshjs(cid:146)clhoelro-gBve.llVeovlue3F7,orm U11n13d,eArTU(cid:148)enrcesrktyainatnyd Di(cid:146)leKuaribstniecsmanan,d(cid:147)JuBidagsmees,n(cid:146)t plinary task such as forecasting an
198i, pp 623-629. Sc,e,uce. VoL i85, i974. pp~ 1i24-113i adversary(cid:146)s intentions.
60
ExpertTeams
The knowledge
requirements for
effective collaboration
The Burden on Intelligence quickly exceed the (cid:149) Pattern bias. In this particular
Analysts example, the data appear to be
capabilities of the
economic and the expert is an
Intelligence is an amalgam of a individual expert. economist. The data are, after all,
number of highly specialized
investment data. Given the back
domains. Within each of these
ground and training of an
domains, a number of experts are economist, it makes perfect sense
tasked with assembling, analyzing,
to try to manipulate the new data
assigning meaning, and reporting on within the context of economics,
sdaotlav,etaheprgooballesm.beoirngmtaokedesacrfiobreec.ast. cdahteamiwsotuolrdbhioelomgeiastnianngdfutlhetroefaore odtehspeirtemotrhee fiamcptotrhtaatntthaenrgelemsa.y he
seek to collaborate with other spe
When an expert encounters a case cialists who might reach different
outside her expertise. her options conclusions regarding the data than (cid:149) 1-leuristic bias. The economist
are to repeat the steps she initially would the economist has spent a career becoming
used to become an expert in the familiar with and using the guid
field. She can In this example, the economist is ing principles of economic
required to use her economic exper analysis and, at best, has only a
(cid:149) Try to make the new data fit with tise in all but the final option of vague familiarity with other
a pattern that she has previously constilting with other experts. In domains and their heuristics. An
stored, the decision to collaborate, the economist would not necessarily
economist is expected to know that know that a chemist or biologist
(cid:149) Recognize that the case falls out what appears to he new economic could identify what substance is
side her expertise and turn to her data may have value to a chemist or being produced based on the
domains hetinstics to try to give biologist, domains with which she types of equipment and supplies
meaning to the data: may have no experience. In other that are being purchased.
words, the economist is expected to
(cid:149) Acknowledge that the case still know that an expert in sonic other
does not fit with her expertise field might find meaning in clara that This example does not describe a
and reject the data set as being an appear to he economic. complex problem(cid:151)most people
would recognize that the data fiom
anomaly, or
Three confounding variables this case might he of value to other
(cid:149) Consult with other experts. affect the economist(cid:146)s domains. Iris one isolated case,
viewed retrospectively, which could
dccisionmaking:
A datum, in and of itself, is not potentially affect two other
domain specific. Imagine eco (cid:149) Processing tune, or context. This domains. l3ut what if the econo
nonlic data that reveal that a does not refer to the amount of mist had to deal with one hundred
country is investing in technologi time necessary to accomplish a data sets per day? Now, multiply
cal infrastructure, chemical task, but rather the moment in those one hundred data sets by the
number of potential domains that
supplies, and research and devel time during which a task
opment. An economist might oiccurs(cid:151).real time(cid:151)and the limi would be interested in any given
decide that the data fit an existing tations that come from being economic cIata set Finally, put all
spending pattern and integrate close to an event. (cid:145)l(cid:146)he econo of this in the context of (cid:147)real time.(cid:148)
these facts with prior knowledge mist doesn(cid:146)t have apiloit The economic expert is now
about a country(cid:146)s economy. The knowledge that the new data set expected to maintain expertise in
same economist might decide that is the critical data set for some economics, which isa full time
this is a new pattern that needs to future event. In (cid:147)real tinie.(cid:148) they endeavor, while simultaneously
he remembered (or stored in long- are simply data to be manipu(cid:151) acquiring some level of experience
term memory) for some future use. latecL It is only in retrospect, or in every other domain. Based on
(cid:145)I(cid:146)he economist might decide that long(cid:151)term memory, that the econ(cid:151) these expectations, the knowledge
the data ate outliers of no conse omist can fit the data into a larger requirements for effective collabo
quence and should he ignored. Or, pattern, weigh their value, and ration quickly exceed the
the economist might decide that the assign them meaning capabilities of the individual expert.
61
Expert Teams
(cid:147)
The fact that data are not
sharedbecomes critical
a
issue in cases of analytic
The expert is left dealing with the Another hope is that each expert(cid:146)s
data through the lens of her own en-or. analysis of the data(cid:151)an increase in
expertise She uses her domain spending and the identification of a
heuristics to met}rporate the data specific chemical agent(cid:151)will come
into an existing pattern, store the together at some point The pi(cid:146)ob(cid:151)
data into long(cid:151)term memory as a lem is at wliat point? Prestimablv,
new pattern, or reject the data set someone will get both of these
as an outlier In each of these data sets to lind data specific to her repoi(cid:146)ts somewhere alt}ng the intel
expertise. This would he inoi(cid:146)di(cid:151)
options, the data stop with the ligence chain Of coui(cid:146)se, the
economist instead of heing shared nately time consuming, individual who gets these reports
with an expert in some other may not be able to synthesize the
doiuain The fact that these cIata Second. during the act of scanning inl(cid:146)ormation That person is sub
are not -shared then becomes a criti large data sets, the expert inevita(cid:151) ject to the same three confounding
cal issue in cases of analytic error2(cid:146) Ny won cl be looking for data that variables described earlier pi(cid:146)o(cid:151)
fit within her area of expertise cessing time, pattern bias. and
In hindsight, critics will say that the Imagine a chemist wht) comes heuristic bias, Rather than solving
implications were obvious(cid:151)that the across data that show that a coun(cid:151) the paradox of expertise, the prob(cid:151)
ci(cid:146)isis could have been avoided if tn(cid:146) is investing in technological 1cm has merely been shifted to
the data had been passed to one infrastructure, chemical supplies. someone else in the organization.
specific expei(cid:146)t or another. In real and research and development (the
time(cid:146) however, an expert cannot same data that the economist ana Tn order to avoid shii(cid:146)ting the prob
know which particular data set lyzed in the previous example). lem from nne expert to another, an
would have value for an expert in The chemist recognizes that these actual collaborative teaiii couId he
another domain. are the ingredients necessary for a htult Why not explicitly put the
nation to produce a specific chemi ec(cid:146)onom isl and the chemist together
cal agent, which cnuId lviye a tti work on analyzing data2 The
The Pros andCons ofTeams tnilitarv application or could be utilitarian pi(cid:146)ohl ems witI his strat(cid:151)
benign. (cid:145)l(cid:146)he chemist then meshes
egy are obvious. Not all economic
One obvious solution to the para the data with an existing pattern, problems are chemical and not all
dox of expertise is to assemble an stores the ciata as a new pattern or chemical pm(cid:146)cihlems aie economic,
interdisciplinan(cid:146) team. Why not ignores the data as an anomaly. Each expert would waste an inorcii(cid:151)
simply make all problem areas or nate amount of time Perhaps one
country(cid:151)specific cliita available to a The chemist, however, has no case in one hundred would he
team of experts from a variety of frame of reference regai(cid:146)dling spend applicable to both experts; during
domains? This ought, at least, to ing trends in the country ofinterest the rest of the day, the experts
reduce the pattern and heuristic The chemist cIt es not km w if this w()tildl drift hack to their mciiviclual
biases inherent in relying on only is an increase, a clcci(cid:146)ease, or a domains, in part because that is
one domain static spending pattern(cid:151)answers what they are best at and in part
that the economist could supply just to stay hus)(cid:146).
Ignoring potential security issues, immediately. (cid:145)l(cid:146)here is no i(cid:146)eason
there are practical problems with for the chemist to know if a coun Closer to the real world, the same
this approach. First, each expert try(cid:146)s ability to pri)duce this example may also have social, polit
would have to sift through large chemical agent is(cid:146)(cid:146)~ new phenome ical. historical, and cultural aspects.
ntin, Perhaps the country in Despite an increase in spending on
question has been producing the a specific chemical agent, the coun
2tN1dtIaiLcgM~,K(cid:145)1I1c)uc(cid:146)aknp&acutoiri,mc,kpr(cid:146),accnCyaiip,,/eilri(cid:146)o9,,6(cid:146)h/9Li~~.iIVF,nf,lS5roh,/1i,IetiRL,ioe(ns(cid:145)rdioIn,(cid:151) cdahteamiacraelpaargtenotffsoormyeeanrsoramnadltphaets(cid:151) e tproyliitnicaqluleys,teinonItumraalyly,nostocihaelly, hist )ri(cid:151)
ic,,ia/,ieDLca,ck(cid:146)rc Win(cid:146) Cocci(cid:146),, Jun ,cc&n/ tern of behavioi(cid:146). callv, or otherwise inclined to cisc it
sa\Vvia,ge7., SlOtI(cid:146)) (cid:145)leRtiiOtwjnLnuintianadhL,ittci(cid:146)llciu,gdeudc,c(cid:146) if9i9n1!)(cid:151), in a thi(cid:146)eatening (cid:145)\(cid:145)ay. There niay be
ic ni rItdcica. NY cornell linive,site One hope is that neither c(cid:146)xpert sociaI ciata(cid:151)u navailable to the
PlitkeIsrsu.IiJ1l9g9riuii.n!RDcWtiisiuhills/tetIrSetra,nf/(cid:145)co(cid:146)cIr,(cid:146)id. 1c/Atithso~r,in(cid:151) treats the data set as an anomaly. economist or the chemist(cid:151)indicat(cid:151)
ford University Press, 19621 that both repcii(cid:146)t it as significant. ing that the chemical agent will he
62
ExpertTeams
(cid:147)
In team building,]
creating shared mental
models, similar
or
used for a benign puipose In order knowledge structures, is tn(cid:146)e effect on generating diverse
for collaboration to work, each team viewpoints within a team, hut
would have to have experts horn not a trivial task. requires more organizational struc
many domains working together on ture than does a homogeneous
the same data set. team
Successful teams have very specific Without specific pi(cid:146)ocesses, oiganiz
organizational and structural ing principles, and operational
requirements. An effective team he possible with an air crew or a structures. interdisciplinary teams
requires discrete and clearly stated tank crew, where an tnclividual(cid:146)s will quickly revert to being just a
goals that are shared by each team role is clearly identifiable as part of room full of experts who ulti
member.22 Teams require interde(cid:151) a larger teatil effort(cid:151)like Iandtng a mately drift back to their previous
pendence and accountabilitv(cid:151)the plane or acquiring and firing on a work patterns. That is, tile experts
success of each individual depends target. Creating shared mental (cid:145),(cid:147)ill not I~e a teato at all they will
on the success of the team as a models in an intelligence team is be a group of experts inclividuaIly
whole and the individual success of less likely, given the vague nature working in some general problem
every other team member.~(cid:145) of the goals. the enormity of the space
task. and the diversits(cid:146) of indh(cid:146)iclual
Effective teams require cohesion. expertise. Moreover, the larger the
formal and informal coinmun ca number of team memhers, the Looking to Technology
tion, cooperation. and shared
tnore difficult it is to generate cohe
ecmoednmgtmeaulsntinmcioacdtweirleossn,.,2~oarnsd\iVmchiiolloaerpeckronahotewisloin(cid:151)on. ScioOto(cid:146)l.perCaotmimounnicla-tlietoeni,(cid:146)ogeanneidty can Talhteerrenatairveespottoenmtuilatliftaeccehtneodlorgeiacmasl,
might be facilitated by specific also he a challenge. It hasa posi(cid:151) An Electronic Perfor ance Support
System (EPSS), for example, is a
work practices, creating shared large database, coupled ~vtth expert
emedngtealstnmioctdueil-scs.,oirs nsoitmialatrrikvniaollvtlas(cid:151)k. a(cid:147)shjarCedannMoenn(cid:151)taftl(cid:145)weSircs,(cid:145)delsE iSnalEaxsp,eSrtClo(cid:146)enavmersDec,ci- sdyesctiesmiso,n ianitdeslliAgepnptlyagienngtss,ucahnda sys
Creating shared mental models ma~7 sIiiHtonIu,siMtd/aIatkIi(cid:145)iccn,//gNi,a!Y(cid:148) Na,l,.,aCd(cid:145)varsCetmnee!heapn,EDreeldchae,u,Co0,,(cid:146),i(cid:145)(cid:145)iAoslnlscto/ke,J,utsistp,e,cs(cid:146)s, theematuoseufnutlelgloi:gtel.nceAtprtohibslepomisntm,ight
1983); L Ctichi and3 French, (cid:147)Overcoming however, the notion of an inte
21 1) Cattsvright and A Zander. GroupDi(cid:146)(cid:151) Resistance to Change,(cid:148) I) Caitwright and A, grated EPSS for large complex data
Itct~tt(cc Resecerci,awl (cid:145)The, r~(cid:146) (New York, Zander, eds Op Cit SI l)eutseh, (cid:147)l(cid:146)he Ef
NY(cid:146) Ilarper& Row, 1960), P Pandi. \V Rich fects ofCoope,ration a,nd Ctimpeto on upoti sets is more theory than practice.27
ardson, and H Conner. ftc impact ofGoal (5roupPr,,cess,(cid:148) I), Cat(cid:146)twrigl it(cid:145)and A l.andei,
Settingon (cid:145)team Simulation Expeiiencc,(cid:148) Sun eds, GrotpI,(cid:146)t (cid:145)0001 ,cs, Resect,(cid:146)cl, on.!(cid:145)Ibooty
it/a/ionawlGatomp. Vol 21, No 4, 1990. pp New. Yoik, NY Harper& Row, 19f,0i. L Fes
~Iii-422,J i-lan(cid:146)ey and C llocRgei, (cid:147)improv tinger. (cid:147)I ntorniaI social Commanication.(cid:148) NhIIs, (cid:145) I(cid:145)nwei Rd ati,ins in (cid:145)I hrce(cid:151)Pe rs(cid:146) in
ing Commonicatun wahin a i\lanageria Cartwright and A, Zander, eds Op, Cit D Gioi,ips,(cid:148) I) Cattwiight and A Zanciei, eds,,
\\(cid:145)urkgroo p.(cid:146) itiruotofApp/tedliehc,,(cid:146),o,(cid:146)a/ ,lohnson. R Johnson, A Onis, a,ndSI Sta,nne, Op Cit L N]olin, (cid:147)Linking Power Structure
,cc(cid:146)u(cid:146)nce, Viil 7. 197), pp 164(cid:151)17~(cid:146) (cid:147)The tmpact ofPtisii (cid:147)c Ct(cid:145)a I atid Resoutee and Pow,er Use,(cid:148) K Cook. ed ,Soc,c,t12v(cid:151)
2~ 5~ l)ei,itsch, (cid:147)rho Effects ofCooperation interdependence on AchIevement, Interac chcotpt(cid:146) iiieori(cid:146) (Newliui-v Park, CA, Sage.
and Competition upon Giotip Proces(cid:146),,(cid:148) t) tion, and Attitodes,(cid:148) ui/riitt!ofCeoera!Psi(cid:148) 1987), V Nieva, E iuleishnian, and A Ricck,
Cartwnghi and A Zandei. edt Op Co D cho/og)(cid:146), Vol 118, No 4 1996, pp 341-3(cid:146)i7. B icc,,,) Ditile(cid:146) t,c,iins (cid:145)flaet rh/c,itt/i(cid:146), (cid:145)ibm,(cid:146) 1Ira(cid:151)
johnson and 1/ Johnson. (cid:147)TJIc ,Iniernal -Dv(cid:151) MullenandC Copper, (cid:147)The Relation Between soI(cid:146)r(cid:146)iutr(cid:146)i Ii, ci~/c/ 17,,ei,(cid:146) kr/ctti,,is/i,pc, RN 85(cid:151) I2
naniics ofCoopeiat (cid:145)-c Leaining Gioups.(cid:146) R Cioup Cohesiveness and Pertormaore An iAlexandria. VA US Aria)(cid:146) Research tnstitoie
Slavin, S sharan, S Kagan, R i(cid:146)iertz-Lazarow Integiation,(cid:148) Psivlto!ogtcc,/!lnI!etio, Vol 115, forthe Behavioral anti Social Sciences, 1985);
its, C Wchh, anti R Schmuck. eds., Jeorii,iIp 599s, pp 210-227,W Ni1hofand I(cid:146) Kommer(cid:146), C Simnicl, TheSoc,o/ogi(cid:146) ofGcorpSuo,,ie!, K
toCooperate Coofvv(cid:146)cili(cid:146)ig I,(cid:146) Leant fNc(cid:146).v (cid:147)An Analysis ofCnopenition in Relation to Wolff, ttans (Clencoc, II, Fiec Press, 1950),
Yoik, NY Plenum, 1985). i) iohnsnn, C, Cognitive Controversy,(cid:148) R Slavin, S Sharati, ~(cid:147) H It)hi(cid:146)istrln, Decttin,,AFaking(cid:146) our!!(cid:145)eifi it
Niaruya va, R Johnson, i) Nelson, and L S Kagan, R tiertz(cid:151)l.a(cid:146),arowii,, C Weiil(cid:146),, and n/c,litr(cid:146) I;nor,n 7i(cid:146)coii c Re.sc(cid:146)c,rc/i Rrtciel/s (Ai
Skon. (cid:147)Effects ofCooperative. Competitive, H schmuck, ella Lea,(cid:146),i nip to Cooperate, Co(cid:151) hngton, VA Defense Advatit,ecl Research
and Intl vidualisaeGoal Statctore on Achieve(cid:151) opera/lop /0Lea,(cid:146),,, (New York, NY, Plenum, ProicctsAgency. 1997), l~ Stealer, (cid:147)individual
tlient- A Nieta(cid:151)Analvsis,(cid:148) l(cid:146)c)(cid:146)c(cid:146)ho/oplcat Bit/tu 1995), j Orasano. (cid:147)Shaied MentalModelsantI i(cid:146)erceptions nfTcarii Le.irning Expeiicnces
to,, Vol 89, No 1, 1981, pp, 47-62, R, Slavin. Cien(cid:146) l(cid:146)crfoi mance.(cid:148) Paper Ptesentcd al the Using Video(cid:151)Based or Viriual Reality Environ(cid:151)
(cid:145)Research on Cooperative i.eatning Consen 3~ith annual meeting ofthe Homan Factors mcnts,(cid:146)(cid:146) D,sserto/to, (cid:145)i tic/inc/sui/en ia/toito?,
si ,s aon Contros(cid:146)ersv,(cid:148) Ec/atccitto,Ia!Leader(cid:151) Sot,ietv, Oilandn, FL, 1990. S Seashore, UNU No 9965200, 2000,
chip, Vol 47. No, 4,1989, pp 52-55,8 Slavin, Goup cohenlet,ncsin theJo.1,t,strial Work(cid:151) 27 H Johnstot,, (cid:145)(cid:145)P iectt(cid:146)ori Ic i(cid:146)ertoi tat,nec Sup
Coopcoviots(cid:146) Tearot,p tNew York. NY Litng(cid:151) group iAnn Arhor, Michigan. Sit University port Sysici,is anti loftirmation Navigatioti
nun, 1983) ofMichigan l(cid:146)res(cid:146),, t954) ihneac/, Vol 2, Nt) 2, 1991, pp 5.7
63
Expert Teams
(cid:147)
Computational systems
have demonstrated
utility in identifying
Ignoring questions about the tech reach conclusions or make discos(cid:146)
nological feasibility of such a patternswithinnarrowly cries Rather than specializing in a
system. Fundamental epistemologi(cid:151) defined and highly specific substantive topic within
cal flaws present imposing hurdles their domain, these experts special
constrained domains;
It is virtually inconceivable that a ize in mastering the reseai(cid:146)ch and
comprehensive computational sys however, inteffigence analytic methods of their domain.
tein could by(cid:151)pass the three
analysis is neither
confounding variables of expertise In the biological and medical fields,
described earlier. narrowly defmed nor these methodological specialists ai(cid:146)e
highly constrained. epidemiologists. In education and
An EPSS. or any other coniputa public policy, these specialists are
tional solution, is designed. program evaluators, In other fields,
programmed, and implemented by they are research methodologists or
a human expert from one domain: statisticians. Despite the label, each
computer science. Historians will field recognizes that it requires
not design the (cid:147)historical decision ate it by reducing all data to a experts in methodology to main
aid,(cid:148) economists will not program binary state. tain and pass on the doimtin(cid:146)s
the (cid:145)economic intelligent agent, heuristics for problem solving and
chemists will not create the chemi (cid:145)l(cid:146)his is not simply a Luddite reac making discoveries.
cal agent expert system (cid:145) Software tion to technology. Computational
engineers and computer scientists svstetns have had a remarkable, The methodologist(cid:146)s focus is on
will do all )f that. positive effect on processing time, selecting and employing a process
storage, and retrieval, They have or processes to research and ana
Computer scientists may consult also demonstrated utiLity in dentify~ lyze data. Specifically, the
with various experts during the ing patterns within narrowly methoclologist identifies the
design phase of such a system, but defined antI highly constrained reseai(cid:146)ch design, the methods For
when it is time to sit down and domains. However, intelligence choosing samples, and the tools For
write code, the programmer will analysis is neither narrowly defined darn analyses. This specialist
follo\v the heuristics of computer nor highly constrained. Quite the becomes an in-house consultant for
science The flexibility, adaptabil opposite, it is multivariate and selecting the process by which one
tv. complexity. and usability of the highly complex, which is why it derives meaning from the data, icc(cid:151)
computational system will he dic requires the expertise of so many ognizes patterns, and solves
tated by the guidelines and rules of diverse fields of study Intelli problems within a domain. Meth(cid:151)
computer science2H In essence. gence analysis is not something a odologists become organizing
one would he trading the heuris computational system handles well. agents within their field by focus
tics from dozens of domains for the While an EPSS, or some other form ing on the heuristics of their
rules that govern computer sci of computational system. may be a domain and validating the method
ence This would reduce the useful tool for maniptilating data, it of discovery for their discipline.
problem of processing time by sim(cid:151) is not a solution to the paradox of
pIifying and linking data, and it expertise. The methodologist holds a unique
may potentially reduce pattern bias. position within the discipline Orga
But it will not reduce heuristic nizing agents are often called on by
bias.22 If anything, it may exagger(cid:151) Analytic Methodologists substantive experts to advise on a
variety of process issues within their
Most domains have specialists who field because they have a different
R nhn%tonand.! Fletcher. 4 tie/cf(cid:151)A mi/ins study the scientific process or perspective than do the experts. On
cj(cid:146)the kfli(cid:146)cii, sazess ofCtimiipiiit(cid:146)r-Bce.cec/ 7)riimi(cid:151) research methods of their disci any given day. an epidemiologist.
inngsotriAt1utteItftotrirD~e(cid:146)fe)on,scetniAciriioa,l,vs(esA,texit9i9n8d),u. VA pline. These people are concerned for example. may be asked to con
2(cid:146)(cid:146) J PieRher and R Johnston. (cid:145)(cid:145)i3ffeetivenes with the epistemology of their sult on studies of the effects of
and Cost Benefits ofComputer-Based DeLi domain, not just philosophically but alcoholisoi on a community or the
sion Aiidi,sHf,oiriE/cql/uiiBpem/(cid:146)ce,nritoMI:uIn(cid:145)ot/enIaSn,c2e0,0(cid:148) 2.Gop,p,, practically. They want to know spread of a virus, or to review a
7/7-728 how experts in their discipline double(cid:151)blind clinical trial of a new
64
ExpertTeams
Few analytic methods
specific to the domain of
intelligence analysis exist
pharmaceutical product. In each the paradox of expertise- Domain
case, the epidemiologist is not being experts aie needed for describing,
asked about the content of the explaining, and problem solving;
study; rather he is being asked to yet, they are not especially good at
comment on the research methods experts in a wide variety of quanti forecasting because the patterns
and data analysis techniques used. tative and qualitative methods. In they m(cid:146)ecognize are limited to their
specific fields of study. They mcvi
other fields~, the methodologists
Well over 200 analytic methods. may he narrowly focused(cid:151)a laho tably look at the world through the
most from domains outside intelli ratory-based experimental lens of their own domain(cid:146)s
gence, arca(cid:148) ailable to the methodologist, for example, or stat heuristics.
intelligence analyst; however, few
istician- In all cases, however,
methods specific to the domain of niethodologists can only he effec \X(cid:146)hat is needed to overcome the
intelligence analysis exist 3(cid:176) Intelli tive if they are experts at the paradox of expertise is a combined
gence analysis lacks specialists process of making meaning ~vithin approach that includes formal the
whose professional ti(cid:146)aining is in their own disciplines matic teams with structured
the process of employing and uni organizational principles; techno
fying the analytic practices within In order to overcome heuristic Logical systems designed with
the field of intelligence. Knowing biases, intelligence agencies need significant input from domain
how to apply methods, select one to focus personnel, resources, and experts; and a cadre of analytic
method over another, weigh dispar training on developing intelligence methodologmsts. Intelligence agen
ate variables, and synthesize the methodologists. These methoclolo cies continue to experiment with
results is left to the individual intel gists will act as in-house the right composition, structure,
ligence analysts(cid:151)the same analysts consultants for analytic teams, gen and organization of analytic teams;
whose expertise is confined to spe erate new methods specific to they budget significant resources
cific substantive areas and their intelligence analysis, modify and for technological solutions; hut
own domains heuristics. improve existing methods of analy comparatively little is being done to
sis, and increase tile advance methodological science.
Intelligence needs methoclologists professionalization of the clisci
to help strengthen the domain of pline of intelligence. Advances in methodology are pri
analysis. Such methodologists need
marily left to the individual
to specialize in the processes that domains- But relying on the sepa
the intelligence domain holds to be Conclusion rate domains risks falling into the
valid. In some fields, like epiclemi same paradoxical trap that cur
ology and program evaluation. Intelligence analysis uses a wide rently exists. What is needed is an
methodologists are expected to he variety of expertise to address a intelligence-centric approach to
multivariate and complex world. methodology, an approach that will
Each expert uses his or her own include the methods and proce
3a(cid:176)nEdxcPeophicmnunns iNneclwudWea,yfst FtoedAenr,alyFzAeCPTolIiOtiNcSs,(cid:148) heuristics to address a small por dures of many domains and tile
H Westerheid. ed I,ms/dcCIA(cid:145)sPneatc \rorhl tion of that world. Intelligence development of heuristics~ ~~ii~ci tech
,
(New Haven, CN- Yale University Press. professionals have the perception niques unique to intelligence. In
A1n9a9l5)y,siRs (HWeausehri,ngPisoync.hoolcogycoefnItne/rclf/o(cid:231)grentchee that somehow all of that disparate short, intelligence analysis needs its
Study ofintelligence. 1999), R Hopkins. analysis will come together at some own analytic heuristics designed,
U(cid:146)an,ii,gs oJ(cid:146)Reeohmtion A CaseStucli(cid:146) ofEl point, either at the analytic team developed, and tested by profes
Sat,mlor,(cid:148) TR 8u-100012 (Washington, DC
Center mr the Study ofInteHigence), j Lock level, through the reporting hierar sional analytic methodologists.
wood and K Lockwood, (cid:147)The Lockwood An chy, or through some This v~(cid:146)ill require using methodolo
aDlcyfncc~ailseMIenttehloldiRefnocrePJroeudri#cmtait,on\(cid:145)o(iLAM3P,)N,o(cid:148) 2, computational aggregation. gists from a variety of other
i994, pp 47-74,J Pierce, (cid:147)some Mathernati domains and professional associa
calMethodsfortntcHigenceAnalysis,(cid:148) S/art,es The intelligence analyst is affected tions at first. hut, in time, the
i1n9I(mdmeicellahsgsr(cid:146)nicfci,ed)S;uEmmSearp,p,Vo(cid:147)Dl,ec2i1s,io1n977T,repeps,(cid:148) 1- by the same confounding variables discipline of analytic methodology
Studies inIntelligence, Winter. Vol. 18, 1974, that affect every other expert: pro will mature into its own sub-disci
Tphpeo4r5-e5m7f(odrecilnatseslilfiigeedn)c,ejAnZailoytsniisc,k(cid:148),H(cid:147)BayWees(cid:146)st cessing time, pattern bias, and pline with its own measures of
ertield, ed.. Op. Cit. heuristic bias. This is the crux of validity and reliability
65