ebook img

DTIC ADA525552: Integrating Methodologists into Teams of Substantive Experts PDF

10 Pages·0.57 MB·English
Save to my drive
Quick download
Download
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.

Preview DTIC ADA525552: Integrating Methodologists into Teams of Substantive Experts

Reducing Analytic Error Integrating Methodologists into Teams of Substantive Experts Rob Johnston Intelligence analysis, like other energy necessary to generate new complex tasks, demands consider knowledge in that field. It takes a able expertise. It requires considerable amount of time and individuals who can recognize pat regular exposure to a large num terns in large data sets, solve ber of cases to become an expert. complex problems, and make pre dictions about future behavior or An individual enters a field ofstudy events. To perform these tasks suc as a novice. The novice needs to (cid:147) cessfully, analysts must dedicate a learn the guiding principles and considerable number of years to rules(cid:151)the heuristics and con Paradoxically, it is the researching specific topics, pro straints(cid:151)of a given task in order cesses, and geographic regions. to perform that task. Concurrently. specificity of expertise the novice needs to be exposed to that makes expert Paradoxically, it is the specificity of specific cases, or instances, that test forecasts unreliable. expertise that makes expert fore the boundaries of such heuristics. casts unreliable. While experts Generally, a novice will find a men outperform novices and machines tor to guide her through the in pattern recognition and problem process of acquiring new knowl solving, expert predictions of future edge. A fairly simple example behavior or events are seldom as would be someone learning to play accurate as simple actuarial tables. chess. The novice chess player In part, this is due to cognitive seeks a mentor to teach her the biases and processing-time con object of the game, the number of straints. In part, it is due to the spaces, the names ofthe pieces, the nature of expertise itself and the function of each piece, how each process by which one becomes an piece is moved, and the necessary expert.1 conditions for winning or losing the game. Becoming an Expert In time, and with much practice, the novice begins to recognize pat Expertise is commitment coupled terns of behavior within cases and, with creativity. Specifically, it is the thus, becomes a journeyman. With commitment of time, energy, and more practice and exposure to resources to a relatively narrow increasingly complex cases, the field of study and the creative journeyman finds patterns not only within cases but also between cases. More importantly, the jour More than 200 individuals contributed to neyman learns that these patterns this study The author is indebted to the re often repeat themselves over time. Dr. RobJohnston is a stheearSchteurdsy.offelilnotwesl,liagenndces,tatfhfeatIntshteitcuteentfeorr Dfocr- The journeyman still maintains reg pthoestCdIocAtoCreanlterresfeoarrtchhefSetlulodwy aotf gttneecnn.s,ceeaAnAndsaslAoycNsiSeasEt,iRotnih,necENvaiSttdiaeofnfncaaelndBMiaslstietudadreyRnetissnetaaetrlcltihh,e supleacricfiocntparcotbwlietmhsaamnednlteoarrntomosroleve CIA University, theJoint Military intelligence complex strategies. Returning to Intelligence and a member of the college, the Naval Postgraduate school, Co the example of the chess player, research staff at the Institute for alnumdbiYaaleUniUvneirvseirtsyi,tyGealosrogceotnotwrnibuUntievdertsoittyh.e the individual begins to learn pat Defense Analyses. project. terns of opening moves, offensive 57 Report Documentation Page Form Approved OMB No. 0704-0188 Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. 1. REPORT DATE 3. DATES COVERED 2003 2. REPORT TYPE 00-00-2003 to 00-00-2003 4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER Integrating Methodologists into Teams of Substantive Experts 5b. GRANT NUMBER 5c. PROGRAM ELEMENT NUMBER 6. AUTHOR(S) 5d. PROJECT NUMBER 5e. TASK NUMBER 5f. WORK UNIT NUMBER 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION Center for the Study of Intelligence,Central Intelligence REPORT NUMBER Agency,Washington,DC,20505 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S) 11. SPONSOR/MONITOR’S REPORT NUMBER(S) 12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release; distribution unlimited 13. SUPPLEMENTARY NOTES Studies in Intelligence, Volume 47, No. 1, 2003 14. ABSTRACT 15. SUBJECT TERMS 16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF 18. NUMBER 19a. NAME OF ABSTRACT OF PAGES RESPONSIBLE PERSON a. REPORT b. ABSTRACT c. THIS PAGE Same as 9 unclassified unclassified unclassified Report (SAR) Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std Z39-18 Expert Teams Apprenticeship. is stifi a standard method of training for many and defensive game-playing strate complex tasks. longer it takes to build expertise, gies, and patterns of victory and or, more accurately, the longer it defeat. takes to experience and store a large number of cases or patterns When a journeyman starts to make and test hypotheses about future behavior based on past expefi The Power ofExpertise ences, she begins the next transition Once she creatively gen instances, however, there are tens, Experts are individuals with special erates knowledge. rather than if not hundreds, of thousands of ized knowledge suited to perform simply matching superficial pat potential patterns. A research study the specific tasks for which they are terns, she hecomes an expert At discovered that chess masters had trained. hut that expertise does not this point, she is confident in her spent between 10.000 and 20,000 necessanly transfer to other knowledge and no longer needs a hours, or more than ten years, domains. A master chess player mentor as a guide(cid:151)she becomes studying and playing chess. On cannot apply chess expertise in a responsible for her own knowl average, a chess master stores. game of poker(cid:151)although both edge. In the chess example. once a 50,000 different chess patterns in chess and poker are games. a chess journeyman begins competing long-term memory.2 master who has never played poker against experts, makes predictions is a nuvice poker player. Similarly, based on patterns, and tests those Similarly, a diagnostic radiologist a biochemist is not qualified to per predictions against actual behavior, spends eight years in full time med form netirostirgery, even though she is generating new knowledge ical training(cid:151)four years of medical both biocheniists and neurosur(cid:151) and a deeper understanding of the school and four years of resi geons study human physiology- In game. She is creating her own dency(cid:151)before she is qualified to other words, the more complex a cases rather than relying on thc take a national board exam and task is, the more specialized and cases of others. begin independent practice.3 exclusive is the knowledge According to a 1988 study, the recluired to perform that task. The chess example is a i(cid:146)ather short average diagnostic radiology resi description of an apprenticeship dent sees forty cases per day, oi(cid:146) An expert perceives meaningful model. Apprenticeship may seem around 12,000 cases per year.5 At patterns in her domain better than like a restrictive 18th century mode the end of a residency, a diagnos non-experts. Where a novice pci(cid:146) of education, but it is still a stan tic radiologist has stored, on ceives random or disconnected data dard method of training for many average, 48,000 cases in long turin points, an expert connects regular complex tasks. Academic doctoral memory. patterns within and between cases programs are based on an appren This ability to identify patterns is ticeship model, as are fields like Psychologists and cognitive scien not an innate percepttial skill; law, music, engineering, and medi tists agree that the time it takes to rather it reflects the organization of cine. Graduate students enter fields become an expert depends on the knowledge after exposure to and of study, find mentors, and begin complexity of the task and thu experience with thousands of the long process of becoming inde number of cases, or patterns, to cases.6 pendent experts and generating which an individual is exposed. new knowledge in their respective The more complex the task, the domains. Ni Niinskv and S Papen A,1,jiczaIIn/c/li(cid:151) genoa(Eugene, OR Oregon Staie System oi Higher Education, 1974). I \(cid:145)nss and 1(cid:146), Pnsi. To some, playing chess may appear 2W chase and H Simon, (cid:147)Perception in (cid:147)On ihe 5,,Jving ofill-Siruc-inred Prohienis,(cid:148) rather trivial when compared, for cphpess5,5(cid:146)-81Cogiiilii(cid:146)e Psychnlngi(cid:146). \roi 4, 1973 NiC)clAiik,inR. G1linais/ce(cid:146)tr.c,raj(cid:146)~n-idrcMIi(cid:145)tFealr,r,,readtsKu. Onpo(cid:145)IcCc/dge example, with making medical American College ufRadiology Personal (london Pinn, 19801. D Egan and B diagnoses, but both are highly com cnmmt.inicaiion, 2002 Schwaaz (cid:147)cirinking in Recall ofSymbolic plex tasks. Chess has a well- GlAaseLre,sgDoidK,lopHferR,uhainndsoYn,WaPngF,eli(cid:147)Eoxvpiecrht.isKe in Di9r7a9w,inpgps,(cid:148)i-ii9/-ain5i8o.n(cid:146)KanNidcKCeoigi;l,iein/,iouJ, Rveoliim7a,n, defined set of heuristics, whereas a Compiex SkiH Diagnosing x-Ray Pictures,(cid:146) H Rueier. and S Hirile. (cid:147)Knowledgc Organi meenddiecdalanddiavganroisaeblse.seeInmbmootrhe open NlliuhraeuClnio~if,AFsRrstpoGecriiraaistseeers,.(Haiinl9ld8s8dN)aiie.FanN(cid:146),J edLsaw,rTenhceeME,r zg1ar9ta8im1u.inneparpnsd,(cid:148)30SkC7io.lg3lu2uD5,itfnfe(cid:145)erPesnic(cid:146)ce(cid:146)hosloigni(cid:146),Comvpoulteir~,Pro(cid:151) 58 ExpertTeams (cid:147) An actuarial table is as good, better, than or an expert at making calls Experts have a deeper understand about the future. ability of experts to make fore ing of their domains than novices casts (cid:176) The performance of do, and utilize higher-order princi experts has been tested against ples to solve problems.7 A novice, actuarial tables to determine if the)(cid:146) for example, might group objects are better at making predictions together by color or size, whereas than simple statistical models. Sev an expert would group the same enty veai(cid:146)s later, with moi(cid:146)e than objects according to their function two hundred experiments in differ omarbelauetnsiiliwntiyg.thofEdxidpfaefterartesanntcdocrmwipteerireigahhewvniatdrhiitnhe Biicenetsstt,earnecxaeptserswethlsfe-arrmeeontmihtoeoryreihnaagwvtaehracenoomnfmoivt eaenqntusawdleormaamiisonunsno,t.6itofisIfdcasltueaparpalbtihoeaudtttwahietpharan tEhxepierrtdsomraeicnosgnibeztetevrartihaabnlensovtihactes. tperdobelrermo.r1s2orEfxaipleerdtstochuencdkertshteainrd a tgiocoudla.rocrasbee,ttaern,atchtauanriaanl etxapbleertisatas have the largest influence on a par solutions more often than novices making calIs about the futu c. atitctuelnatriopnroobnletmhosaendvalroiacbulsest.heir ainngd irnefcoorgmnaitzieonwhneecnestshaeryyafroer miss sEpveecnifiifcacnaseexpienrftorimsagtiivoenntmhoanreis available to the statistical model, solving a problem(cid:146)3 Experts are Experts have better domain-spe aware of the limits of their domain tfhoermextpheertacdtuoaersianlottabtleen.d(cid:145)7 to outper knowledge and apply their cific short-term and long-term domain(cid:146)s heuristics to solve prob moveemr,orexypetrhtasnpneorvfiocresm dtoas.k(cid:148) s Minortheeir lexepmesritheantcefalblaosuetside of their Trehseeraercahrefifnedwingesx,cebputtiotnhse teoxctheepse domains faster than novices and c1commmsiotlvifnegw.e9r eInrtreorresstwihnigllye,perxopebrts The Paradox ofExpertise etixopnesrtasrearienfgoirvmeantitvhee. reWshuletns of the go about solving problems differ ently than novices. Experts spend (cid:145)l(cid:146)he strengths of expertise can also (cid:145)~ FL Reichenhach, EsperienccandPrethctitu, more time thinking about a prob he weaknesses)(cid:146) Although one (cid:145)chicago, IL University ofChicago i(cid:146)ress, lem to fully understand it at the would expect experts to he good 1938), 1 Sarbin, (cid:147)A ConirilautiontoiheSisidy ofAciuarialand individual Med odsofPreclic(cid:151) beginning of a task than do nov forecasters, they are not particu lion,(cid:148) AniencanJoz,rnalofSoc~ologs(cid:146), Vol 18, ices, who immediately seek to find larly good at making predictions 1943, pp 593-602 a solution.(cid:146)0 Experts use their about the future. Since the 1930s, ViSergsuDsaAwcetso.ai0ialiJ(cid:146)auusdig,meanntd,(cid:148)aScMieee,,hcl,,,V(cid:147)colli.ni2c4a3l. knowledge of previous cases as researchers have been testing the i989, pp~ i668-i674; W Grove and i(cid:146), Mcclii, context for creating mental models !Comparanve Efficiency ofinformal IStibiec live, impressionistic) and Formal (Mechani to solve given problems. cal, Algorithmic) Piediction Procedures(cid:146) The v0,~5 and T Posi, (cid:147)On the Solving ofill clinical-Statistical Coniro~(cid:146)ersy,(cid:148) Psicholog)., Struciured Problems(cid:148) Ni Chi, R Glaser, and Pu/thePolw~(cid:146). andan(cid:146), \(cid:145)ol, 2, No 2, 1996, Ni Farr. eds, Op Cil pp 293-323 NI Chi, P Peliovich, and R Glaser, !caiego iaNi Chi. H Glaser, and E Rees, Expertisein i7 R Dawes, (cid:147)A Case Study ofGraduate Ad nzat,on and Represeniaiion ofPhysics Prob ProblemSolving.(cid:148) R Siernberg.ed Advances missions Applicaiion ofThree Principles of lems by Experts and Novices,(cid:148) Cognitive ui/hePs)(cid:146)cholopj(cid:146) nj/lumen Intell.igence Human Decision Making, A,nencanJ(cid:146)sychol Science. VoL 5, i98i, pp~ i21-125. NI, We,ser (H,llsdale. NJ Lawrence Erlbauni Associ ogist, voL 26, i971, pp. 180-188; \V, Grove and J sheria. Programming Problem Repre aies.i982); 0 Simon and H Simon, (cid:145)individ and P NIeehl, Op. Co (see fooinoie 16), H sentation in Novice and Expert Program ualDifferences inSolving PhysicsProblems,(cid:148) Sacks, (cid:147)Promises, Performance, and Princi mers.(cid:146) lnctrucltoi;al/onrnalo,filfasi(cid:151)iiIach,ne R Siegler. ed,, Children~Thinking, WhatDc ples An empirical said)(cid:146) ofParole Decision- Studies, vol 14, 1983, pp 39i-396 ir/ops(cid:146) (Hillsdale, NJ(cid:146) Lawrence Erlhaum As making in Connecticut,(cid:148) Co,z,,ect,cutfan,Re W chase and K Ericsson, (cid:147)Skill and Work sociates, 1978) v/en(cid:146), Vol 9. 1977, pp 349-422, 1 Sarhin, (cid:147)A ing Memory,(cid:148) G Bower, ed ThePsi(cid:146)chologr 3 Larkin, The Role ofProblem Represenia Conirihution io the Siudy ofActuarial and in oj(cid:146)Leanii;ig a,,dA!oti,(cid:146)atio;, (, NewYork, NY t,on in Physics,(cid:148) 0 Gentner and A Sievens, dividual Methods ofPrediction,(cid:148) A,oencan Academic Press. 1982). eds Men/a/Modelsti-tillsdale, NJ La(cid:146)~(cid:146)rence JoiirnalofSoctologi(cid:146), 1943, pp 18, 593-602,J ow, Chase. (cid:147)Spatial Represcniaiions ofTaxi Erlb,aum Associates, 1983). Sawyer, (cid:147)Measurement and Piedicuon, Clini Drivers.(cid:148) 0. Rogers andJ, Slohada. eds.. Ac ii C, Camerer and E Johnson, (cid:147)The Process- cal and Statistical,(cid:148) Pc)(cid:145)chologica/ Riillelin, qziisitioo oJSs(cid:146)nibohc Skills (New York, NY(cid:146) Performance Paradox in Expert Judgment Vol 66, 1966, pp 178-200,W, SchofieldandJ Plenum. 1983). Ho(cid:148) Can Experts KnowsoMuchandPredict Garrard. Longiiudinal Study ofMedical Sir, ~5J PaigeandH, Simon, (cid:147)cognition i(cid:146)rocesses so Badly(cid:148) K Ericsson and J Smith, eds, To dents Selected forAdmission io Medical in SolvingAlgebra \Vord Problems,(cid:146) B Klein ,,(cid:145)ard aGeneralTheoryofK~penise Prospects SchoolbyAciuarialandCom,nitieeMethods,(cid:146) muniz, ed Prohlr(cid:146)ni Solving(NewYork, NY nullLuoit(cid:145)cambridge. UK Cambridge Un, 13r1,shJournaloJ(cid:146)ilethcal Education, Vol 9, Wiley, 1966,(cid:146) vcrsiiy Press, 199i) 1975. pp 86-90 59 Expert Teams (cid:147) Forecasting is complex, a interdisciplinary, actuarial predictions, for example, dynamic, and (cid:149) Pattern bias(cid:151)looking for cvi they tend to score as well as the dence that confirms rather than statistical model if they use the sta multivariate task. rejects a hypothesis and inadvert tistical information in making their ently filling in missing data with own predictions.(cid:145)~ In addition, if data from prevkuis experiences. an expert has privileged informa tion that is not reflected in the (cid:149) Heuristic bias(cid:151)using inappropri statistical table, she will actually perform hetter than the table. A Theorists and researchers differ aptreedgiuctiidoenlsi.nes or rules to make classic example is the broken leg when trying to explain why experts argument: Judge X has gone to the are less accurate forecasters than theater every Friday night for the statistical models. Some have The very method by which one past ten years. Based on an actuar argued that experts. like all becomes an expert explains why ial table, one would predict, with humans, are inconsistent when experts are much better at describ some certainty, that the ludge using mental models to make pre ing, explaining, performing tasks. would go to the theater this Friday dictions. That is. the model an and problem(cid:151)solving ~vithin their night. An expert knows, however, expert uses for predicting X in one domains than are novices, but, with that the judge broke her leg Thurs month is different from the model a few exceptions, are worse at fore day afternoon and is currently in used for predicting X in a follow casting than actuarial tables based the hospital until Saturday. Know ing month, although precisely the on historical, statistical models ing this key variable allows the same case and same data set are expert to predict that the judge will used in both instances. ~ A given domain has specific heui(cid:146)is not attend the theater this Friday tics for performing tasks~~~incI night. A number of researchers point to solving problems. These rules are human biases to explain unreliable a large part of what makes up Although this argument makes expert predictions During the last expertise. In addition, experts sense, it is misleading. Forecasting 30 years, researchers have catego need to acquire and store tens of is not simply a linear logical argu rized, experimented, and theorized thousands of cases within their ment hut rather a complex, about the cognitive aspects of fore domains in order to recognize pat interdisciplinary, dynamic, and mul casting.2(cid:176) Despite such efforts, the terns, generate and test hypotheses, and contribute to the collective tivariate task. Cases are rare where literature shows little consensus knowledge within their fields. In woeniegkheeydvaaprpiraobplreiaisteklnyotwondeatnedr rtieognasrdoifnghutmheancabuisaess oNronmeatnhiefleesstsa. other words, becoming an expert mine an outcome. Generally, no there is general agreement that two yreeaqrusireosf vaiseiwginnifgictahnetwnourmlbdetrhroofugh single static variable predicts types of bias exist: the lens of one specific domain. It behavior; rather, many dynamic is the specificity that gives the variables interact, weight and value expert the power to recognize pat change, and other variables are J~ Fries, c(cid:146)t at, (cid:147)Assessmeni ofRadiologic terns, perform tasks, and solve introduced or omitted to determine Pdorooigirzeesds,iocnonitnrRolhleeudmaTntaoli.d(cid:148) AA,rtlhhrnittiissRhAeu,Rnain problems. outcome. as written by author vol 29, No i, i98fi, pp 1-9 2e4)sjanEdvanCso,nsBeiqauseinnceHsiu(nHao,v,e,ReaU.KconiLna8wreCnacues Paradoxically, it is this same speci L Goldberg, (cid:145)simple Models orsimple Pro ErlbaumAssociates, i989), R Iteuer, Pci(cid:146)chot ficity that is restrictive, narrowly cesses? some Research on clinicalJudg ogyoJlnteltigeiuceAuualis/s(Washington, DC focusing the experts attention on merns,(cid:148) AinencanPsi(cid:146)cbo/ogist. VoL 23, i968, cenierfortheStudyofintelligence. i999), D pp 483-496, L Goldberg, (cid:147)Man versusModel Kahneman, P Slovic. and A. Tversky,Ju,d8- one domain to the exclusion of ofMan A Rationale, Plus some Evidence, for uncut UnderUncertain/i lieunsticsandBi others. It should come as little sur a Method ofimproving on clinical tnfereoc ases cambridge, UK(cid:146) cambridge University prise, then. that an expert would Cs,(cid:148) Pssr.holag,ealBulletin, Vol 73, 1970. pp Press. 1982): A Tversky and D Kahnemao, 422-432, ft Leli and 5. Filskov, (cid:147)clinical-Ac (cid:147)The Beliefin the Law ofsmall Numbers,(cid:146)(cid:146) have difficulty identifying and tuarial Detection ofand Description ofBrain PsychologicalBulletin, Vol 76, i97i, pp i05- weighing variables in an interdisci it,m(cid:148)Jpoauirrnmae/ntowfiCtlh/utih/ecaWlePcshjs(cid:146)clhoelro-gBve.llVeovlue3F7,orm U11n13d,eArTU(cid:148)enrcesrktyainatnyd Di(cid:146)leKuaribstniecsmanan,d(cid:147)JuBidagsmees,n(cid:146)t plinary task such as forecasting an 198i, pp 623-629. Sc,e,uce. VoL i85, i974. pp~ 1i24-113i adversary(cid:146)s intentions. 60 ExpertTeams The knowledge requirements for effective collaboration The Burden on Intelligence quickly exceed the (cid:149) Pattern bias. In this particular Analysts example, the data appear to be capabilities of the economic and the expert is an Intelligence is an amalgam of a individual expert. economist. The data are, after all, number of highly specialized investment data. Given the back domains. Within each of these ground and training of an domains, a number of experts are economist, it makes perfect sense tasked with assembling, analyzing, to try to manipulate the new data assigning meaning, and reporting on within the context of economics, sdaotlav,etaheprgooballesm.beoirngmtaokedesacrfiobreec.ast. cdahteamiwsotuolrdbhioelomgeiastnianngdfutlhetroefaore odtehspeirtemotrhee fiamcptotrhtaatntthaenrgelemsa.y he seek to collaborate with other spe When an expert encounters a case cialists who might reach different outside her expertise. her options conclusions regarding the data than (cid:149) 1-leuristic bias. The economist are to repeat the steps she initially would the economist has spent a career becoming used to become an expert in the familiar with and using the guid field. She can In this example, the economist is ing principles of economic required to use her economic exper analysis and, at best, has only a (cid:149) Try to make the new data fit with tise in all but the final option of vague familiarity with other a pattern that she has previously constilting with other experts. In domains and their heuristics. An stored, the decision to collaborate, the economist would not necessarily economist is expected to know that know that a chemist or biologist (cid:149) Recognize that the case falls out what appears to he new economic could identify what substance is side her expertise and turn to her data may have value to a chemist or being produced based on the domains hetinstics to try to give biologist, domains with which she types of equipment and supplies meaning to the data: may have no experience. In other that are being purchased. words, the economist is expected to (cid:149) Acknowledge that the case still know that an expert in sonic other does not fit with her expertise field might find meaning in clara that This example does not describe a and reject the data set as being an appear to he economic. complex problem(cid:151)most people would recognize that the data fiom anomaly, or Three confounding variables this case might he of value to other (cid:149) Consult with other experts. affect the economist(cid:146)s domains. Iris one isolated case, viewed retrospectively, which could dccisionmaking: A datum, in and of itself, is not potentially affect two other domain specific. Imagine eco (cid:149) Processing tune, or context. This domains. l3ut what if the econo nonlic data that reveal that a does not refer to the amount of mist had to deal with one hundred country is investing in technologi time necessary to accomplish a data sets per day? Now, multiply cal infrastructure, chemical task, but rather the moment in those one hundred data sets by the number of potential domains that supplies, and research and devel time during which a task opment. An economist might oiccurs(cid:151).real time(cid:151)and the limi would be interested in any given decide that the data fit an existing tations that come from being economic cIata set Finally, put all spending pattern and integrate close to an event. (cid:145)l(cid:146)he econo of this in the context of (cid:147)real time.(cid:148) these facts with prior knowledge mist doesn(cid:146)t have apiloit The economic expert is now about a country(cid:146)s economy. The knowledge that the new data set expected to maintain expertise in same economist might decide that is the critical data set for some economics, which isa full time this is a new pattern that needs to future event. In (cid:147)real tinie.(cid:148) they endeavor, while simultaneously he remembered (or stored in long- are simply data to be manipu(cid:151) acquiring some level of experience term memory) for some future use. latecL It is only in retrospect, or in every other domain. Based on (cid:145)I(cid:146)he economist might decide that long(cid:151)term memory, that the econ(cid:151) these expectations, the knowledge the data ate outliers of no conse omist can fit the data into a larger requirements for effective collabo quence and should he ignored. Or, pattern, weigh their value, and ration quickly exceed the the economist might decide that the assign them meaning capabilities of the individual expert. 61 Expert Teams (cid:147) The fact that data are not sharedbecomes critical a issue in cases of analytic The expert is left dealing with the Another hope is that each expert(cid:146)s data through the lens of her own en-or. analysis of the data(cid:151)an increase in expertise She uses her domain spending and the identification of a heuristics to met}rporate the data specific chemical agent(cid:151)will come into an existing pattern, store the together at some point The pi(cid:146)ob(cid:151) data into long(cid:151)term memory as a lem is at wliat point? Prestimablv, new pattern, or reject the data set someone will get both of these as an outlier In each of these data sets to lind data specific to her repoi(cid:146)ts somewhere alt}ng the intel expertise. This would he inoi(cid:146)di(cid:151) options, the data stop with the ligence chain Of coui(cid:146)se, the economist instead of heing shared nately time consuming, individual who gets these reports with an expert in some other may not be able to synthesize the doiuain The fact that these cIata Second. during the act of scanning inl(cid:146)ormation That person is sub are not -shared then becomes a criti large data sets, the expert inevita(cid:151) ject to the same three confounding cal issue in cases of analytic error2(cid:146) Ny won cl be looking for data that variables described earlier pi(cid:146)o(cid:151) fit within her area of expertise cessing time, pattern bias. and In hindsight, critics will say that the Imagine a chemist wht) comes heuristic bias, Rather than solving implications were obvious(cid:151)that the across data that show that a coun(cid:151) the paradox of expertise, the prob(cid:151) ci(cid:146)isis could have been avoided if tn(cid:146) is investing in technological 1cm has merely been shifted to the data had been passed to one infrastructure, chemical supplies. someone else in the organization. specific expei(cid:146)t or another. In real and research and development (the time(cid:146) however, an expert cannot same data that the economist ana Tn order to avoid shii(cid:146)ting the prob know which particular data set lyzed in the previous example). lem from nne expert to another, an would have value for an expert in The chemist recognizes that these actual collaborative teaiii couId he another domain. are the ingredients necessary for a htult Why not explicitly put the nation to produce a specific chemi ec(cid:146)onom isl and the chemist together cal agent, which cnuId lviye a tti work on analyzing data2 The The Pros andCons ofTeams tnilitarv application or could be utilitarian pi(cid:146)ohl ems witI his strat(cid:151) benign. (cid:145)l(cid:146)he chemist then meshes egy are obvious. Not all economic One obvious solution to the para the data with an existing pattern, problems are chemical and not all dox of expertise is to assemble an stores the ciata as a new pattern or chemical pm(cid:146)cihlems aie economic, interdisciplinan(cid:146) team. Why not ignores the data as an anomaly. Each expert would waste an inorcii(cid:151) simply make all problem areas or nate amount of time Perhaps one country(cid:151)specific cliita available to a The chemist, however, has no case in one hundred would he team of experts from a variety of frame of reference regai(cid:146)dling spend applicable to both experts; during domains? This ought, at least, to ing trends in the country ofinterest the rest of the day, the experts reduce the pattern and heuristic The chemist cIt es not km w if this w()tildl drift hack to their mciiviclual biases inherent in relying on only is an increase, a clcci(cid:146)ease, or a domains, in part because that is one domain static spending pattern(cid:151)answers what they are best at and in part that the economist could supply just to stay hus)(cid:146). Ignoring potential security issues, immediately. (cid:145)l(cid:146)here is no i(cid:146)eason there are practical problems with for the chemist to know if a coun Closer to the real world, the same this approach. First, each expert try(cid:146)s ability to pri)duce this example may also have social, polit would have to sift through large chemical agent is(cid:146)(cid:146)~ new phenome ical. historical, and cultural aspects. ntin, Perhaps the country in Despite an increase in spending on question has been producing the a specific chemical agent, the coun 2tN1dtIaiLcgM~,K(cid:145)1I1c)uc(cid:146)aknp&acutoiri,mc,kpr(cid:146),accnCyaiip,,/eilri(cid:146)o9,,6(cid:146)h/9Li~~.iIVF,nf,lS5roh,/1i,IetiRL,ioe(ns(cid:145)rdioIn,(cid:151) cdahteamiacraelpaargtenotffsoormyeeanrsoramnadltphaets(cid:151) e tproyliitnicaqluleys,teinonItumraalyly,nostocihaelly, hist )ri(cid:151) ic,,ia/,ieDLca,ck(cid:146)rc Win(cid:146) Cocci(cid:146),, Jun ,cc&n/ tern of behavioi(cid:146). callv, or otherwise inclined to cisc it sa\Vvia,ge7., SlOtI(cid:146)) (cid:145)leRtiiOtwjnLnuintianadhL,ittci(cid:146)llciu,gdeudc,c(cid:146) if9i9n1!)(cid:151), in a thi(cid:146)eatening (cid:145)\(cid:145)ay. There niay be ic ni rItdcica. NY cornell linive,site One hope is that neither c(cid:146)xpert sociaI ciata(cid:151)u navailable to the PlitkeIsrsu.IiJ1l9g9riuii.n!RDcWtiisiuhills/tetIrSetra,nf/(cid:145)co(cid:146)cIr,(cid:146)id. 1c/Atithso~r,in(cid:151) treats the data set as an anomaly. economist or the chemist(cid:151)indicat(cid:151) ford University Press, 19621 that both repcii(cid:146)t it as significant. ing that the chemical agent will he 62 ExpertTeams (cid:147) In team building,] creating shared mental models, similar or used for a benign puipose In order knowledge structures, is tn(cid:146)e effect on generating diverse for collaboration to work, each team viewpoints within a team, hut would have to have experts horn not a trivial task. requires more organizational struc many domains working together on ture than does a homogeneous the same data set. team Successful teams have very specific Without specific pi(cid:146)ocesses, oiganiz organizational and structural ing principles, and operational requirements. An effective team he possible with an air crew or a structures. interdisciplinary teams requires discrete and clearly stated tank crew, where an tnclividual(cid:146)s will quickly revert to being just a goals that are shared by each team role is clearly identifiable as part of room full of experts who ulti member.22 Teams require interde(cid:151) a larger teatil effort(cid:151)like Iandtng a mately drift back to their previous pendence and accountabilitv(cid:151)the plane or acquiring and firing on a work patterns. That is, tile experts success of each individual depends target. Creating shared mental (cid:145),(cid:147)ill not I~e a teato at all they will on the success of the team as a models in an intelligence team is be a group of experts inclividuaIly whole and the individual success of less likely, given the vague nature working in some general problem every other team member.~(cid:145) of the goals. the enormity of the space task. and the diversits(cid:146) of indh(cid:146)iclual Effective teams require cohesion. expertise. Moreover, the larger the formal and informal coinmun ca number of team memhers, the Looking to Technology tion, cooperation. and shared tnore difficult it is to generate cohe ecmoednmgtmeaulsntinmcioacdtweirleossn,.,2~oarnsd\iVmchiiolloaerpeckronahotewisloin(cid:151)on. ScioOto(cid:146)l.perCaotmimounnicla-tlietoeni,(cid:146)ogeanneidty can Talhteerrenatairveespottoenmtuilatliftaeccehtneodlorgeiacmasl, might be facilitated by specific also he a challenge. It hasa posi(cid:151) An Electronic Perfor ance Support System (EPSS), for example, is a work practices, creating shared large database, coupled ~vtth expert emedngtealstnmioctdueil-scs.,oirs nsoitmialatrrikvniaollvtlas(cid:151)k. a(cid:147)shjarCedannMoenn(cid:151)taftl(cid:145)weSircs,(cid:145)delsE iSnalEaxsp,eSrtClo(cid:146)enavmersDec,ci- sdyesctiesmiso,n ianitdeslliAgepnptlyagienngtss,ucahnda sys Creating shared mental models ma~7 sIiiHtonIu,siMtd/aIatkIi(cid:145)iccn,//gNi,a!Y(cid:148) Na,l,.,aCd(cid:145)varsCetmnee!heapn,EDreeldchae,u,Co0,,(cid:146),i(cid:145)(cid:145)iAoslnlscto/ke,J,utsistp,e,cs(cid:146)s, theematuoseufnutlelgloi:gtel.nceAtprtohibslepomisntm,ight 1983); L Ctichi and3 French, (cid:147)Overcoming however, the notion of an inte 21 1) Cattsvright and A Zander. GroupDi(cid:146)(cid:151) Resistance to Change,(cid:148) I) Caitwright and A, grated EPSS for large complex data Itct~tt(cc Resecerci,awl (cid:145)The, r~(cid:146) (New York, Zander, eds Op Cit SI l)eutseh, (cid:147)l(cid:146)he Ef NY(cid:146) Ilarper& Row, 1960), P Pandi. \V Rich fects ofCoope,ration a,nd Ctimpeto on upoti sets is more theory than practice.27 ardson, and H Conner. ftc impact ofGoal (5roupPr,,cess,(cid:148) I), Cat(cid:146)twrigl it(cid:145)and A l.andei, Settingon (cid:145)team Simulation Expeiiencc,(cid:148) Sun eds, GrotpI,(cid:146)t (cid:145)0001 ,cs, Resect,(cid:146)cl, on.!(cid:145)Ibooty it/a/ionawlGatomp. Vol 21, No 4, 1990. pp New. Yoik, NY Harper& Row, 19f,0i. L Fes ~Iii-422,J i-lan(cid:146)ey and C llocRgei, (cid:147)improv tinger. (cid:147)I ntorniaI social Commanication.(cid:148) NhIIs, (cid:145) I(cid:145)nwei Rd ati,ins in (cid:145)I hrce(cid:151)Pe rs(cid:146) in ing Commonicatun wahin a i\lanageria Cartwright and A, Zander, eds Op, Cit D Gioi,ips,(cid:148) I) Cattwiight and A Zanciei, eds,, \\(cid:145)urkgroo p.(cid:146) itiruotofApp/tedliehc,,(cid:146),o,(cid:146)a/ ,lohnson. R Johnson, A Onis, a,ndSI Sta,nne, Op Cit L N]olin, (cid:147)Linking Power Structure ,cc(cid:146)u(cid:146)nce, Viil 7. 197), pp 164(cid:151)17~(cid:146) (cid:147)The tmpact ofPtisii (cid:147)c Ct(cid:145)a I atid Resoutee and Pow,er Use,(cid:148) K Cook. ed ,Soc,c,t12v(cid:151) 2~ 5~ l)ei,itsch, (cid:147)rho Effects ofCooperation interdependence on AchIevement, Interac chcotpt(cid:146) iiieori(cid:146) (Newliui-v Park, CA, Sage. and Competition upon Giotip Proces(cid:146),,(cid:148) t) tion, and Attitodes,(cid:148) ui/riitt!ofCeoera!Psi(cid:148) 1987), V Nieva, E iuleishnian, and A Ricck, Cartwnghi and A Zandei. edt Op Co D cho/og)(cid:146), Vol 118, No 4 1996, pp 341-3(cid:146)i7. B icc,,,) Ditile(cid:146) t,c,iins (cid:145)flaet rh/c,itt/i(cid:146), (cid:145)ibm,(cid:146) 1Ira(cid:151) johnson and 1/ Johnson. (cid:147)TJIc ,Iniernal -Dv(cid:151) MullenandC Copper, (cid:147)The Relation Between soI(cid:146)r(cid:146)iutr(cid:146)i Ii, ci~/c/ 17,,ei,(cid:146) kr/ctti,,is/i,pc, RN 85(cid:151) I2 naniics ofCoopeiat (cid:145)-c Leaining Gioups.(cid:146) R Cioup Cohesiveness and Pertormaore An iAlexandria. VA US Aria)(cid:146) Research tnstitoie Slavin, S sharan, S Kagan, R i(cid:146)iertz-Lazarow Integiation,(cid:148) Psivlto!ogtcc,/!lnI!etio, Vol 115, forthe Behavioral anti Social Sciences, 1985); its, C Wchh, anti R Schmuck. eds., Jeorii,iIp 599s, pp 210-227,W Ni1hofand I(cid:146) Kommer(cid:146), C Simnicl, TheSoc,o/ogi(cid:146) ofGcorpSuo,,ie!, K toCooperate Coofvv(cid:146)cili(cid:146)ig I,(cid:146) Leant fNc(cid:146).v (cid:147)An Analysis ofCnopenition in Relation to Wolff, ttans (Clencoc, II, Fiec Press, 1950), Yoik, NY Plenum, 1985). i) iohnsnn, C, Cognitive Controversy,(cid:148) R Slavin, S Sharati, ~(cid:147) H It)hi(cid:146)istrln, Decttin,,AFaking(cid:146) our!!(cid:145)eifi it Niaruya va, R Johnson, i) Nelson, and L S Kagan, R tiertz(cid:151)l.a(cid:146),arowii,, C Weiil(cid:146),, and n/c,litr(cid:146) I;nor,n 7i(cid:146)coii c Re.sc(cid:146)c,rc/i Rrtciel/s (Ai Skon. (cid:147)Effects ofCooperative. Competitive, H schmuck, ella Lea,(cid:146),i nip to Cooperate, Co(cid:151) hngton, VA Defense Advatit,ecl Research and Intl vidualisaeGoal Statctore on Achieve(cid:151) opera/lop /0Lea,(cid:146),,, (New York, NY, Plenum, ProicctsAgency. 1997), l~ Stealer, (cid:147)individual tlient- A Nieta(cid:151)Analvsis,(cid:148) l(cid:146)c)(cid:146)c(cid:146)ho/oplcat Bit/tu 1995), j Orasano. (cid:147)Shaied MentalModelsantI i(cid:146)erceptions nfTcarii Le.irning Expeiicnces to,, Vol 89, No 1, 1981, pp, 47-62, R, Slavin. Cien(cid:146) l(cid:146)crfoi mance.(cid:148) Paper Ptesentcd al the Using Video(cid:151)Based or Viriual Reality Environ(cid:151) (cid:145)Research on Cooperative i.eatning Consen 3~ith annual meeting ofthe Homan Factors mcnts,(cid:146)(cid:146) D,sserto/to, (cid:145)i tic/inc/sui/en ia/toito?, si ,s aon Contros(cid:146)ersv,(cid:148) Ec/atccitto,Ia!Leader(cid:151) Sot,ietv, Oilandn, FL, 1990. S Seashore, UNU No 9965200, 2000, chip, Vol 47. No, 4,1989, pp 52-55,8 Slavin, Goup cohenlet,ncsin theJo.1,t,strial Work(cid:151) 27 H Johnstot,, (cid:145)(cid:145)P iectt(cid:146)ori Ic i(cid:146)ertoi tat,nec Sup Coopcoviots(cid:146) Tearot,p tNew York. NY Litng(cid:151) group iAnn Arhor, Michigan. Sit University port Sysici,is anti loftirmation Navigatioti nun, 1983) ofMichigan l(cid:146)res(cid:146),, t954) ihneac/, Vol 2, Nt) 2, 1991, pp 5.7 63 Expert Teams (cid:147) Computational systems have demonstrated utility in identifying Ignoring questions about the tech reach conclusions or make discos(cid:146) nological feasibility of such a patternswithinnarrowly cries Rather than specializing in a system. Fundamental epistemologi(cid:151) defined and highly specific substantive topic within cal flaws present imposing hurdles their domain, these experts special constrained domains; It is virtually inconceivable that a ize in mastering the reseai(cid:146)ch and comprehensive computational sys however, inteffigence analytic methods of their domain. tein could by(cid:151)pass the three analysis is neither confounding variables of expertise In the biological and medical fields, described earlier. narrowly defmed nor these methodological specialists ai(cid:146)e highly constrained. epidemiologists. In education and An EPSS. or any other coniputa public policy, these specialists are tional solution, is designed. program evaluators, In other fields, programmed, and implemented by they are research methodologists or a human expert from one domain: statisticians. Despite the label, each computer science. Historians will field recognizes that it requires not design the (cid:147)historical decision ate it by reducing all data to a experts in methodology to main aid,(cid:148) economists will not program binary state. tain and pass on the doimtin(cid:146)s the (cid:145)economic intelligent agent, heuristics for problem solving and chemists will not create the chemi (cid:145)l(cid:146)his is not simply a Luddite reac making discoveries. cal agent expert system (cid:145) Software tion to technology. Computational engineers and computer scientists svstetns have had a remarkable, The methodologist(cid:146)s focus is on will do all )f that. positive effect on processing time, selecting and employing a process storage, and retrieval, They have or processes to research and ana Computer scientists may consult also demonstrated utiLity in dentify~ lyze data. Specifically, the with various experts during the ing patterns within narrowly methoclologist identifies the design phase of such a system, but defined antI highly constrained reseai(cid:146)ch design, the methods For when it is time to sit down and domains. However, intelligence choosing samples, and the tools For write code, the programmer will analysis is neither narrowly defined darn analyses. This specialist follo\v the heuristics of computer nor highly constrained. Quite the becomes an in-house consultant for science The flexibility, adaptabil opposite, it is multivariate and selecting the process by which one tv. complexity. and usability of the highly complex, which is why it derives meaning from the data, icc(cid:151) computational system will he dic requires the expertise of so many ognizes patterns, and solves tated by the guidelines and rules of diverse fields of study Intelli problems within a domain. Meth(cid:151) computer science2H In essence. gence analysis is not something a odologists become organizing one would he trading the heuris computational system handles well. agents within their field by focus tics from dozens of domains for the While an EPSS, or some other form ing on the heuristics of their rules that govern computer sci of computational system. may be a domain and validating the method ence This would reduce the useful tool for maniptilating data, it of discovery for their discipline. problem of processing time by sim(cid:151) is not a solution to the paradox of pIifying and linking data, and it expertise. The methodologist holds a unique may potentially reduce pattern bias. position within the discipline Orga But it will not reduce heuristic nizing agents are often called on by bias.22 If anything, it may exagger(cid:151) Analytic Methodologists substantive experts to advise on a variety of process issues within their Most domains have specialists who field because they have a different R nhn%tonand.! Fletcher. 4 tie/cf(cid:151)A mi/ins study the scientific process or perspective than do the experts. On cj(cid:146)the kfli(cid:146)cii, sazess ofCtimiipiiit(cid:146)r-Bce.cec/ 7)riimi(cid:151) research methods of their disci any given day. an epidemiologist. inngsotriAt1utteItftotrirD~e(cid:146)fe)on,scetniAciriioa,l,vs(esA,texit9i9n8d),u. VA pline. These people are concerned for example. may be asked to con 2(cid:146)(cid:146) J PieRher and R Johnston. (cid:145)(cid:145)i3ffeetivenes with the epistemology of their sult on studies of the effects of and Cost Benefits ofComputer-Based DeLi domain, not just philosophically but alcoholisoi on a community or the sion Aiidi,sHf,oiriE/cql/uiiBpem/(cid:146)ce,nritoMI:uIn(cid:145)ot/enIaSn,c2e0,0(cid:148) 2.Gop,p,, practically. They want to know spread of a virus, or to review a 7/7-728 how experts in their discipline double(cid:151)blind clinical trial of a new 64 ExpertTeams Few analytic methods specific to the domain of intelligence analysis exist pharmaceutical product. In each the paradox of expertise- Domain case, the epidemiologist is not being experts aie needed for describing, asked about the content of the explaining, and problem solving; study; rather he is being asked to yet, they are not especially good at comment on the research methods experts in a wide variety of quanti forecasting because the patterns and data analysis techniques used. tative and qualitative methods. In they m(cid:146)ecognize are limited to their specific fields of study. They mcvi other fields~, the methodologists Well over 200 analytic methods. may he narrowly focused(cid:151)a laho tably look at the world through the most from domains outside intelli ratory-based experimental lens of their own domain(cid:146)s gence, arca(cid:148) ailable to the methodologist, for example, or stat heuristics. intelligence analyst; however, few istician- In all cases, however, methods specific to the domain of niethodologists can only he effec \X(cid:146)hat is needed to overcome the intelligence analysis exist 3(cid:176) Intelli tive if they are experts at the paradox of expertise is a combined gence analysis lacks specialists process of making meaning ~vithin approach that includes formal the whose professional ti(cid:146)aining is in their own disciplines matic teams with structured the process of employing and uni organizational principles; techno fying the analytic practices within In order to overcome heuristic Logical systems designed with the field of intelligence. Knowing biases, intelligence agencies need significant input from domain how to apply methods, select one to focus personnel, resources, and experts; and a cadre of analytic method over another, weigh dispar training on developing intelligence methodologmsts. Intelligence agen ate variables, and synthesize the methodologists. These methoclolo cies continue to experiment with results is left to the individual intel gists will act as in-house the right composition, structure, ligence analysts(cid:151)the same analysts consultants for analytic teams, gen and organization of analytic teams; whose expertise is confined to spe erate new methods specific to they budget significant resources cific substantive areas and their intelligence analysis, modify and for technological solutions; hut own domains heuristics. improve existing methods of analy comparatively little is being done to sis, and increase tile advance methodological science. Intelligence needs methoclologists professionalization of the clisci to help strengthen the domain of pline of intelligence. Advances in methodology are pri analysis. Such methodologists need marily left to the individual to specialize in the processes that domains- But relying on the sepa the intelligence domain holds to be Conclusion rate domains risks falling into the valid. In some fields, like epiclemi same paradoxical trap that cur ology and program evaluation. Intelligence analysis uses a wide rently exists. What is needed is an methodologists are expected to he variety of expertise to address a intelligence-centric approach to multivariate and complex world. methodology, an approach that will Each expert uses his or her own include the methods and proce 3a(cid:176)nEdxcPeophicmnunns iNneclwudWea,yfst FtoedAenr,alyFzAeCPTolIiOtiNcSs,(cid:148) heuristics to address a small por dures of many domains and tile H Westerheid. ed I,ms/dcCIA(cid:145)sPneatc \rorhl tion of that world. Intelligence development of heuristics~ ~~ii~ci tech , (New Haven, CN- Yale University Press. professionals have the perception niques unique to intelligence. In A1n9a9l5)y,siRs (HWeausehri,ngPisoync.hoolcogycoefnItne/rclf/o(cid:231)grentchee that somehow all of that disparate short, intelligence analysis needs its Study ofintelligence. 1999), R Hopkins. analysis will come together at some own analytic heuristics designed, U(cid:146)an,ii,gs oJ(cid:146)Reeohmtion A CaseStucli(cid:146) ofEl point, either at the analytic team developed, and tested by profes Sat,mlor,(cid:148) TR 8u-100012 (Washington, DC Center mr the Study ofInteHigence), j Lock level, through the reporting hierar sional analytic methodologists. wood and K Lockwood, (cid:147)The Lockwood An chy, or through some This v~(cid:146)ill require using methodolo aDlcyfncc~ailseMIenttehloldiRefnocrePJroeudri#cmtait,on\(cid:145)o(iLAM3P,)N,o(cid:148) 2, computational aggregation. gists from a variety of other i994, pp 47-74,J Pierce, (cid:147)some Mathernati domains and professional associa calMethodsfortntcHigenceAnalysis,(cid:148) S/art,es The intelligence analyst is affected tions at first. hut, in time, the i1n9I(mdmeicellahsgsr(cid:146)nicfci,ed)S;uEmmSearp,p,Vo(cid:147)Dl,ec2i1s,io1n977T,repeps,(cid:148) 1- by the same confounding variables discipline of analytic methodology Studies inIntelligence, Winter. Vol. 18, 1974, that affect every other expert: pro will mature into its own sub-disci Tphpeo4r5-e5m7f(odrecilnatseslilfiigeedn)c,ejAnZailoytsniisc,k(cid:148),H(cid:147)BayWees(cid:146)st cessing time, pattern bias, and pline with its own measures of ertield, ed.. Op. Cit. heuristic bias. This is the crux of validity and reliability 65

See more

The list of books you might like

Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.