OPERATIONS ASSESSMENT IN AFGHANISTAN IS BROKEN What Is to Be Done? Stephen Downes-Martin I n the absence of a credible numbers-based theory of counterinsurgency there can be no objective, numbers-based assessment for Operation ENDURING FREEDOM.TheU.S.militarynonethelesshasattemptedtoconductanumbers- basedassessmentprocess.Thus,whenanewcommanderandstafftakeoverdu- ties as a regional command in Afghanistan,they inherit an operations assess- mentprocessriddledwithhighlyvisibleflawsthatemanatefromtheimproper useofnumbersandflawedlogic.Whilenoassessmentprocesscanbeperfector free of any criticism, the flaws the author observed during a six-week stint in-countryaresufficientlyegregiousthattheyseriouslyreducethevaluethose assessments provide to commanders’ decision sup- Dr.StephenDownes-Martin,researchprofessoratthe NavalWarCollege,hasoverthirtyyearsofexperience port. In addition, the visibility of these flaws means indevelopingandapplyingwargaming,gametheory, thatmilitaryassessments,andbyassociationthemili- decisionanalysis,andsystemsthinkingtotactical,oper- ational,andstrategicdecisionsupport.HehasaPhDin tary commanders,are rightfully distrusted by higher physicsfromKing’sCollege,UniversityofLondon,was civilian authority and by other organizations within amilitaryintelligenceofficerintheBritishArmy,andis the theater. It is therefore imperative that incoming nowaU.S.citizen.ThecommanderofIMarineExpedi- tionary Force (Forward) invited him to be his assess- commandersandstaffstakingoverresponsibilitiesfor mentsadviserforthespringof2010,duringwhichtime regional commands address these flaws to improve heworkedwith,observed,andinterviewedofficers,senior decisionmakingandtoearnthetrustofhighercivil- noncommissionedofficers,andciviliansatavarietyof assessments-relatedstaffsatregionalcommands,provin- ianauthorityandorganizationswithwhomtheyhave cialreconstructionteams,theInternationalSecurityAs- towork. sistance Force Headquarters, and the International StaffsandcommandersinAfghanistancreatedop- SecurityAssistanceForceJointCommandHeadquarters. Thisarticleisaproductofthatworkandofdiscussions erations assessment processes under extraordinarily withofficersandanalystsduringtheyearsincehisvisit. difficult circumstances while fighting, and it is ex- traordinary how well they have done given those Naval War College Review, Autumn 2011, Vol. 64, No. 4 Report Documentation Page Form Approved OMB No. 0704-0188 Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. 1. REPORT DATE 3. DATES COVERED 2011 2. REPORT TYPE 00-00-2011 to 00-00-2011 4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER Operations Assessment in Afghanistan is Broken: What Is to Be Done? 5b. GRANT NUMBER 5c. PROGRAM ELEMENT NUMBER 6. AUTHOR(S) 5d. PROJECT NUMBER 5e. TASK NUMBER 5f. WORK UNIT NUMBER 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION Naval War College,686 Cushing Road,Newport,RI,02841-1207 REPORT NUMBER 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S) 11. SPONSOR/MONITOR’S REPORT NUMBER(S) 12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release; distribution unlimited 13. SUPPLEMENTARY NOTES 14. ABSTRACT 15. SUBJECT TERMS 16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF 18. NUMBER 19a. NAME OF ABSTRACT OF PAGES RESPONSIBLE PERSON a. REPORT b. ABSTRACT c. THIS PAGE Same as 23 unclassified unclassified unclassified Report (SAR) Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std Z39-18 104 NAVALWARCOLLEGEREVIEW circumstances.Nevertheless,it is necessary to identify and fix the flaws in the present operations assessment process to strengthen decision support and the validityoftheassessments,withoutapportioningblameorcriticism.Inthisar- ticleIavoididentifyingindividualsororganizationsasmuchaspossibleindis- cussing flaws in the assessments processes that I have observed. Instead, I addresstheproliferationof “junkarithmetic”andflawedlogicwithinthecur- rently used assessment processes and discuss why regional commanders and theirstaffsshouldcareabouttheseproblems,bydescribingthedamagetocom- manders’credibilityanddecisionsupportcreatedbyflawedprocesses.Finally,I propose an approach to operations assessment that regional commanders can immediatelyputintoplace.Idonotdiscussorcommentonstrategy,operations, or the broader arguments concerning counterinsurgency versus counter- terrorism.Ifocussolelyontheoperationsassessmentprocess. Dr. Jonathan Schroden convincingly argues (in an accompanying article) concerningoperationsassessmentthat“thereisanentirefailurecycleatwork, and until its associated issues are rectified,the theoretical promises of opera- tionsassessmentwillcontinuetogounrealized.”Regionalcommandersdonot havetowaitforhighercommandtodealwiththesekeyissues,norshouldthey; they have the authority and capability to deal with them within their own re- gional commands. The assessment approach I propose in this article meshes withtherequestsforinformationfromhighercommandandfitswithintheop- erationalplanningprocessoftheregionalcommand.Itusesmilitaryandcivil- ianprofessionaljudgmentappliedtotheappropriatecombinationofobjective andsubjectivedata,backedupwithvalidarithmeticandsoundlogic. WHAT IS OPERATIONS ASSESSMENT? Jointdoctrinedescribesassessmentas“aprocessthatmeasuresprogressofthe 1 joint force toward mission accomplishment.” Joint doctrine also makes clear thatsimplymeasuringprogressisinsufficient,thattheassessmentprocessmust “help commanders adjust operations and resources as required, determine whentoexecutebranchesandsequels,andmakeothercriticaldecisionstoen- surecurrentandfutureoperationsremainalignedwiththemissionandmilitary 2 endstate.” ImplicationsoftheDoctrinalDefinitionofOperationsAssessment Howexactlydoesonemeasureprogresstowardaccomplishingamission?Un- lessonehasalreadyaccomplishedthemission,missionaccomplishmentorfail- urewilloccurinthefuture,butonedoesassessmentinthepresent,usingpresent andpastinformation.Therefore,bydefinition,operationsassessmentisanat- 3 tempttoforecastfuturesuccessbasedoncurrentandpastexperience. Itattempts DOWNES-MARTIN 105 toprovideinsightintohowwellthecommandisachievingitsoperationalobjec- tives(occurringinthefuture)usinginformationfromtheenvironment(describ- ingthepresentandthepast).Assessmentshouldincludeameasureof thegap betweenthecurrentsituationandthefuturedesiredendstate,therateofclosure (orwidening)ofthatgapandaforecastofitsfuturerateofclosure(orwiden- ing),and an assessment of the risk to the endurance of the end state after the achievementofobjectivesandterminationofmilitaryaction. Therefore,inordertoprovidedecisionsupporttothecommanderwithinthe guidelineslaiddownbyjointdoctrine,operationsassessmentmustanswerwhat Icall“theassessmentquestion,”whichingeneralhastheform:“Whatisthelike- lihoodof,andwhataretherisksto,theconditionsforthespecifiedendstates occurringorremainingstableifmilitaryoperationsareterminatedonthespec- ifieddate?” InthecontextofAfghanistan,Iproposethattheassessmentprocessatthere- gional command level must answer the more specific question: “What is the likelihoodof,andwhataretherisksto,theconditionsforthespecifiedendstates occurringorremainingstableif theregiontransitionsfromcoalitionforcesto fullGovernmentoftheIslamicRepublicofAfghanistan(GIRoA)controlonthe 4 specifieddate?” Iftheoperationsassessmentcellanswersthisquestion,thecommanderand staffwillbeinabetterpositiontoreportprogresstowardtheobjective,iden- tify risks to achieving the objective,and propose reallocation of resources to 5 minimizeormitigatetherisks. Assessinganyindividuallineof operationor objectiveisamatterofansweringtheassessmentquestioninthecontextofthe likelihood that the end states specified for the objectives remain stable after 6 transitionandwhattherisksaretothoseendstatesshouldtransitionoccur. Inmyopinion,linesof operationorobjectivesinAfghanistanareonlyof in- teresttotheextentthattheyenableatransitionofpowertoGIRoAbythespec- ifieddatewiththespecifiedendstatesremainingstable. Forecastinghasalonganddubioushistory,fullofpseudoscience,junkarith- 7 metic,andflawedlogic,practicedbywitchesandlistenedtobykings. Forecast- ing should be done using a combination of subjective professional judgment, 8 objective logic,(social) science,and mathematics. However,for many people the differences between pseudoscience and real science are hard to spot. Fur- thermore,approachesthatarevalidinonecontextcanbecomeinvalidinothers, evenifthedifferencesinthecontextsarenotobvious.Soalthoughmostofficers would subscribe to the notion of using valid logic, mathematics, and science (everyone believes that they themselves are rational and logical),it is difficult for those not explicitly educated and trained in science, analysis, and critical 106 NAVALWARCOLLEGEREVIEW thinkingtoidentifywhetheranapproachislogicallyorscientificallyvalid.This difficultyistherootcauseofmanyoftheflawsIhaveobservedinoperationsas- 9 sessmentaspracticedinAfghanistan. RequirementsforOperationsAssessment Iclaimthatfourprimaryrequirementsmustbemetifanoperationsassessment processistoprovidegooddecisionsupporttoaregionalcommander. Assessments at the different levels of warfare (tactical, operational, strategic) andacrosstheinstrumentsofnationalpowermustbelinked.Linkage“up”thelev- elsofwarfaremeansthattheassessmentsforalldistrictsinaregionshouldplay majorpartsintheregionalassessmentandthattheassessmentsforallregions shouldplaymajorpartsinthenationalassessment.Therewillbeemergentef- fects at each level that are not described by combining the assessments from lowerlevels,butanyassessmentshouldprovidelogicalreasonsforthem,based ontheinputs.Linkage“across”theinstrumentsof nationalpowerprovidesan integrated assessment combining the separate diplomatic, information, mili- tary,andeconomicsassessmentsfromcivilianandmilitarybranchesofthein- volvedcoalitiongovernments. Metricsandoperationalendstatesmustbelogicallyconnected.Theremustbea logicalconnectionbetweentheprocessedmetricsdataandaforecastoftheen- duranceoftheendstatesiftransitionweretooccur—thatis,credibleandlogical reasonswhythe(qualitativeandquantitative)valuesofthemetricsforecastthe stability(orinstability)oftheendstates,shouldtransitionoccur.Satisfyingthis requirement provides a mechanism for addressing the risks to the end states, whichiscriticaltosupportingthecommander’sdecisionsabouttheallocations ofresourcesneededtodealwiththoserisks.Inaddition,thisrequirementhelps determinewhatmetricsarerequired. The appropriate metrics must be identified and the data to process them col- lected.Therearethreepossiblefailuremodeshere:collectingirrelevantmetrics, not collecting necessary metrics,and not knowing which of the previous two failuresarepresent. Themetricsdatamustbeprocessedusingvalidlogic,arithmetic,andscience.In theabsenceofanobjective,numerictheoryofcounterinsurgency,itisespecially necessary that assessment staffs creatively apply subjective, professional judg- menttotheobjectiveandsubjectivemetricdatainordertoanswertheassess- 10 mentquestion. Itiscriticalthatsuchcreativitynotviolateestablishedrulesof logic, mathematics, or science, lest staffs generate unidentified errors when makinganassessmentanddamageanydecisionthatusestheassessment. IarguethatthemilitaryassessmentsIhaveobservedinAfghanistanclearly donotsatisfythesefourrequirements. DOWNES-MARTIN 107 FLAWS IN ASSESSMENT AS CURRENTLY PRACTICED Ihaveidentifiedbydirectobservationsixmajorflawstocredibleandqualityoper- ationsassessments.Theserenderassessmentsunfitforprovidingdecisionsupport tothecommander,therebycastingdoubtonthecredibilityoftheassessmentsand upon the credibility ofthe commands and commanders that use them. Overoptimism An officer must be prepared and able to wear two very different hats.One of theseisthe“plannerandanalyst”hatwornwhen,amongotheroccasions,doing assessments.Theotheristhe“leader”hat,wornwhenleadingsubordinatesin the execution of plans. The first requires a pessimist mind-set (“the glass is half-empty”) and focuses on critical thinking and the application of logic to identifyingandovercomingwhatcangowrong,inordertoidentifyandmitigate risk and, in turn, to ensure mission success. The second requires an optimist mind-set(“theglassishalf-full”)andafocusoninspiringsubordinates,politi- 11 cians,andcivilianstoachieveobjectivesdespitetherisks. Thetwomind-sets (pessimismandoptimism)areverydifferent,buttheformeriscriticaltoassess- ment,forthefollowingreasons. Operationsassessmentrequiresanalysisusinglogicandelementsofthesci- entificmethod.Acriticalcomponentofthescientificmethodistheconceptof “falsifiability”—thatahypothesismustbecapableofbeingdisprovedinorderto beworthwhile.Onedoesnotprovesomehypothesistobetrue;insteadthebest 12 onecandoisfailtodisproveit. Similarly,inapplyingevidentialreasoningto distinguishbetweenalternative explanationsusing“analysisof competinghy- potheses,”onedoesnotcomparethestrengthsofsupportingevidenceforvari- ous alternatives; instead one compares the weaknesses of evidence against 13 14 them. Underlying these concepts is the fact that knowledge is contingent. Thesenotionsarecounterintuitive,buttheyarewellestablished,andtheyun- derlie the last three hundred years of successful Western science and the last 15 2,400yearsofWesternphilosophicalandlogicalthought. Officerswhoweartheirleaderhatswhenanalyzingorassessingriskproduc- ingpooranalysesorassessments,andofficerswhoweartheiranalysthatswhile leadingplaceexecutionatrisk.Inadditionanofficermustbeverycarefulifde- liberatelydecidingtoweara“glasshalf-full”hatwhenreportinganassessment, whetherupthechainofcommand(includingtohighercivilianauthority)or to external organizations, such as the media. The risk here is inappropriate optimism. Inadditiontothenecessarycriticalattitudeforassessment,anofficerwhois requiredtobringcreativesubjectiveassessmenttobearmustbecapableofap- plying inductive logic, which requires a divergent mind-set capable of 108 NAVALWARCOLLEGEREVIEW recognizingpatternsfrompartialinformation.Thedangerhereisthathuman- ityhasevolvedmostlytoerronthesideoffalsepositiveswhenlookingforpat- terns(“bettersafethansorry”),whichinthecaseofanaturallypositiveattitude willleadtoclaimingoptimisticpatternsinthedata. Althoughmilitarycommandersandtheirstaffsworkhardatavoidingover- optimism,theytendtobringtheirleadership(“wecandoit”/“glasshalf-full”) characteristics to bear during analysis and assessment,and there is an institu- tionaldrivetoproduce“good-newsstories.”Thislatterdriveispartiallyinre- sponsetothe“bad-newsstories”reportedinthepress,andpartiallyinresponse totheimperativetoshowprogressintimetoservetheendsofvariouspolitical timetables.Itisextremelydifficult,tothepointofimpossibility,foranindivid- ualtoachievethecorrectpositiveandnegativebalance,butanorganizationcan, anditbestachievesthisbalancebydeliberatelysettingupanadversarialprocess, usingdevil’sadvocacy.Thegoalofthisprocessistoidentifyandexamineallthe waysinwhichthingscangowrong,inordertoinstitutionalizethecritical“glass half-empty”attitudeandensurethatthenaturaldesireforgoodnewstopasson upthechainofcommanddoesnotdominateassessmentorreporting. MetricsCollection Ihaveobservedtwomajortypesofmetricscollectionproblems.Thefirstprob- lem,promiscuousmetricscollection,breaksdownintotwoparts,self-inflicted andinflictedfromabove.Someassessmentscellsandteamstoldmetheycollect asmuchinformationandasmanymetricsastheycanthinkof,“justincase.”In thesecases,thestatedgoalwastobeableto“changewhatweanalyzeasobjec- tivesorrequestsforinformationchangewithouthavingtochangewhatweare collecting.”Thisresultsinahighlikelihoodthatmanyof thecollectedmetrics arenotrelevanttothesituationbeingassessed. In addition, the International Security Assistance Force (ISAF), the ISAF JointCommand(IJC),andsomeregionalcommandshaveissued“fragmentary orders”(FRAGOs,orFRAGORDs),requiringthecollectionoflargenumbersof numericmetrics(see,forexample,theRegionalOperationalDesignEffectsAs- sessment[RODEA]structureusedforawhilebyRegionalCommand[South], 16 describedin figure 1). Itisbeyondthe capacityof mostforcesin the fieldto collectonthelargenumbersofdemandedmetricsandproduceacredibleprod- uct.Frequentattemptsbysubject-matterexpertstoreducethenumberofmet- rics by brainstorming are unlikely to help, for two reasons. First, there is no guarantee that the original lists of metrics contained all those required, so anyreducedlistmaymissnecessaryones;second,itwasmostlybrainstormingin the absence of an analytic framework that gave rise to the problem of metrics DOWNES-MARTIN 109 FIGURE1 “bloat” in the first place. Trying to REGIONAL OPERATIONAL DESIGN EFFECTS ASSESS- correctaproblembydoingthesame MENT(RODEA)STRUCTURE thing that created it is unlikely to succeed. The second problem, blinkered metricscollection,attemptstoavoid the first problem but introduces worse consequences. Some assess- ments cells told me that they try to identify up front which metrics are hard or impossible to collect, and then set these aside.The problem is thatthereisnoanalysisastowhether theseignoredmetricsshouldbecol- lected, and therefore there is a risk that critical metrics will be neither collected nor considered. What is worse, commanders are not in- formedthattheassessmentisignor- ing metrics whose importance has not been determined.This jeopardizes the accuracyoftheassessmentandhidesfromthecommanderandfromlaterasses- sorstheoriginaldecisiontoignorehard-orimpossible-to-collectmetrics.Since theassessmentmayverywellbepoor,thecredibilityofthecommanderisulti- matelyplacedatrisk. JunkArithmetic Usingarithmeticonnumericmetricsisoptional,buttherulesofarithmeticare 17 notoptional. ThefollowingexamplesofjunkarithmeticIencounteredsuffice todemonstratethebroaderproblem. Manyof theassessmentsprocesses Iobservedin-theatertake qualitative and quantitativedata,rankorderthem,andaveragetherank-ordernumbers.Forex- ample,in the RODEA process,assessors coded answers to questions on a point scaleofonethroughfive,similartothe“ratingdefinitionlevels”usedbyISAFand IJC.Thesecodesarenotratio-scalenumbers,andtherefore,bythelawsofarith- metic,functionssuchas“averaging”cannotbeperformedonthem—itwouldbe 18 meaningless. Toputthisintoafamiliarcontext,officerpaygradesarerankor- deredby“Onumber”—thatis,paygradesO-1(secondlieutenant)throughO-10 (four-stargeneral).Butnoonebelievesthatabrigadiergeneral(O-7)isthesame as a major (O-4) paired with a captain (O-3) just because four plus three is 110 NAVALWARCOLLEGEREVIEW 19 seven. Averagingordinalnumbers,suchasrankorders,withinanassessment processisjustasnonsensical,andthiskindofobviouserrorsubjectsthecredi- bilityoftheassessment,andthecommandpromotingit,tojustifiablesuspicion. Valuesformanymetricsareobtainedusingpolls.InAfghanistan,thesepolls haveclaimedmarginsoferrorofapproximatelyplusorminus3percentforna- 20 tionwide surveys,5 percent for regions,and 10 percent for districts. Given a plus-or-minus 10 percent margin of error, a district metric would have to change by approximately 20 percent before one could claim a trend. When a changeislessthanapproximatelydoublethemarginoferror,thesoundestcon- clusionthatcanbedrawnis“Wedonotknowwhethertherehasbeenachange ornot.”UnfortunatelyinmostassessmentsIobservedin-theaterthesemargins oferrorwereignored,andinasignificantnumberofinstancesofficersclaimed unjustifiedtrendsonsmallchangesof data.An“assessmentsdashboard”Iob- served did not even have a symbol for “trend unknown,”just check-boxes for “trendimproving,”“static,”and“declining”;itwasimpossibletoreportthatit wasnotknownwhetherthereevenwasatrend.WhenIaskedaboutthis,theof- ficerinchargereplied,“Themilitarydoesnotliketoadmitwedonotknow,so wereportoneof thedashboardoptionsandthencaveatthereport.”Toreport “Weknowthatthismeasureisnotchanging”isobviouslynotthesameastore- port“Wedonotknowifthismeasureischanging”;thetwosituationshavevery different implications for the commander’s decision making. It is doubtful whetheranyonerememberscaveatsafteratrendreporthasbeendelivered. Anotherobservedexampleofjunkarithmetic(thistimeleadingtoanoverly optimistic claim) was an Afghan National Police assessment claiming that the organization was “nearly 100 percent filled.” Examination of the underlying datashowedthatpatrolmenwereovermannedwhileofficersandnoncommis- sionedofficerswereundermanned,bysignificantamounts.Whetherornotthe assessmentreportedtheunderlyingdataalongwiththeconclusion,thefactre- mainsthat“nearly100percentfilled”simplydidnotfaithfullysummarizethe situationinthiscase.The“nearly100percentfilled”summaryhadbeenderived byapplyingjunkarithmetictotheunderlyingdata,andalltoooftentheunder- lyingdatathatwouldrevealthetruesituationdonotmakeitintothereports. Theuseofanyarithmeticonnumericmetricsincounterinsurgencyissubject tosuspicion,themoresowhenoneattemptstorollupthenumbersintosome grandscoreofhowwellwearedoing.EveninacountryasstableastheUnited States,withalltheeconomicdataandinformationonecouldconceivablyaskfor andnoonetryingtokillyouasyouaskforit,westilldonothavecredibleeco- nomicforecastmodelsthatcanavoidnear-catastrophiceconomicmeltdowns. Wherethenisthecrediblenumbers-basedmodelforgovernance,development, and security in Afghanistan during an insurgency? Certainly, a professional, DOWNES-MARTIN 111 subjective,qualitativeassessmentofprogresswillmakeuseofcertainnumbers, butnotbyrunningarithmeticfunctionsoverthem. SimplisticColorCoding Seniorcommanders’timeisapreciousandnonrenewableresource;staffsright- fully guard it jealously.The most common approach in-theater to providing seniorcommanderswiththeconclusionsof assessmentsistoproduceacolor- codedmap,eachdistrictshowninoneoffivecolorsindicatingthelevelofsuc- 21 cessthere(seefigure2). Butashasbeenpointedoutandpersuasivelyarguedto aworkinggroupofseniorofficers,includinggenerals,atIJC,“Thecolor-coded map dilutes transparency and accuracy and offers a simplistic and misleading 22 representationofthebattlespace.” I have observed that most senior commanders demand narrative explana- tionsfromsubordinatesduringbriefingsandforwardnarrativeassessmentsup thechain.However,staffsdonotusuallycollect,document,orstorethesenarra- tivesinadatabaseformsuitableforlateranalysis.Sincetheseseniorcommanders FIGURE2 EXAMPLEOVERALLDISTRICTASSESSMENTS,COLORCODEDBYLEVELOFSUCCESS ShayrKhanBandar Hairatan TorahGhundey IslamQal'eh TorKham GhulumKhan Zaranj Wesh(Chaman) BahramChah ISAFJointCommandMetricsWorkshop,Washington,D.C.,17–18March2010,unclassifiedhandout.