Modified Adversarial Hierarchical Task Network Planning in Real-Time Strategy Games

24 Pages·2017·3.62 MB·English

applied
sciences
Article
Modified Adversarial Hierarchical Task Network
Planning in Real-Time Strategy Games
LinSun,PengJiao,KaiXu,QuanjunYin*andYabingZha
CollegeofInformationSystemandManagement,NationalUniversityofDefenseTechnology,Changsha410073,
China;[e-mail](L.S.);[e-mail](P.J.);[e-mail](K.X.);
[e-mail](Y.Z.)
* Correspondence:[e-mail];Tel.:[phone]
Received:31July2017;Accepted:20August2017;Published:25August2017
Abstract: Theapplicationofartificialintelligence(AI)toreal-timestrategy(RTS)gamesincludes
considerablechallengesduetotheverylargestatespacesandbranchingfactors,limiteddecision
times,anddynamicadversarialenvironmentsinvolved. Toaddressthesechallenges,hierarchical
tasknetwork(HTN)planninghasbeenextendedtodevelopamethoddenotedasadversarialHTN
(AHTN), and this method has achieved favorable performance. However, the HTN description
employed cannot express complex relationships among tasks and accommodate the impacts of
the environment on tasks. Moreover, AHTN cannot address task failures during plan execution.
Therefore, this paper proposes a modified AHTN planning algorithm with failed task repair
functionalitydenotedasAHTN-R.ThealgorithmextendstheHTNdescriptionbyintroducingthree
additionalelements: essentialtask,phase,andexitcondition. Ifanytaskfailsduringplanexecution,
theAHTN-RalgorithmidentifiesandterminatesallaffectedtasksaccordingtotheextendedHTN
description, and applies a novel task repair strategy based on a prioritized listing of alternative
plans to maintain the validity of the previous plan. In the planning process, AHTN-R generates
the priorities of alternative plans by sorting all nodes of the game search tree according to their
primaryfeatures. Finally,empiricalresultsarepresentedbasedonaµRTSgame,andtheperformance
ofAHTN-RiscomparedtothatofAHTNandtotheperformancesofotherstate-of-the-artsearch
algorithmsdevelopedforRTSgames.
Keywords: HTNplanning;real-timestrategygame;taskrepair
1. Introduction
The application of artificial intelligence (AI) to real-time strategy (RTS) games includes
considerable challenges due to the very large state spaces and branching factors, limited decision
times,anddynamicadversarialenvironmentsinvolved[1]. RTSgamesareregardedasasimplification
ofreal-lifeenvironments,andcouldthereforeserveasatestbedforinvestigatingactivitiessuchas
real-timeadversarialplanninganddecisionmakingunderuncertainty[2]. Moreover,taskplanning
techniquesthathavebeendemonstratedtobeeffectiveinRTSgamescouldalsobeappliedinreal-world
domains[2].
Comparedwithconventionalboardgames,RTSgameshavethefollowingprimarydifferences[2]:
1. Playerscanpursueactionssimultaneouslywiththeactionsofotherplayers,andneednottake
turns, asingamessuchaschess. Moreover, playeractionscanbeconductedoververyshort
decisiontimes,allowingforrapidsequencesofactions.
2. Playerscanpursueconcurrentactionsemployingmultiplecontrollableunits. Thisismuchmore
complexthanconventionalboardgames,whereonlyasingleactionisperformedwitheachturn.
Appl.Sci.2017,7,872;doi:10.3390/app7090872 www.mdpi.com/journal/applsci
Appl.Sci.2017,7,872 2of24
3. Playeractionsaredurative,inthatanactionrequiresnumeroustimestepstobeexecuted. This
isalsomuchmorecomplexthanconventionalboardgames,whereactionsmustbecompleted
withinasingleturn.
4. Thestatespaceandbranchfactorsaretypicallyverylarge.Forexample,atypical128×128mapin
“StarCraft”generallyincludesabout400playercontrollableunits. Consideringonlythelocation
ofeachunit, thenumberofpossiblestatesisabout101685, whereasthestatespaceofchessis
typicallyestimatedtobearound1050. Ofcourse,evenlargervaluesareobtainedwhenincluding
theotherfactorsintheRTSgame.
5. TheenvironmentofanRTSgameisdynamic. Unlikeinconventionalboardgamessuchaschess,
wherethefundamentalnaturesoftheboardandgamepiecesneverchange,theenvironment
ofanRTSgamemaychangedrasticallyinresponsetoplayerdecisions,whichcaninvalidatea
generatedplan.
Becauseofthedifferencesdiscussedabove,standardgametreesearchmethods,whichperform
wellforboardgameslikealpha–betasearch[3],cannotbedirectlyappliedinRTSgames[4]. However,
research has beenconducted to modify game tree searchmethods to addressthese differences [4].
Chungetal.[5]investigatedtheapplicabilityofgametreeMonteCarlosimulationsinRTSgames.
BallaandFern[6]appliedtheupperconfidenceboundfortrees(UCT)algorithminanRTSgame
to address the complications associated with durative actions. The UCT algorithm is a variant of
MonteCarlotreesearch(MCTS)thatmodifiesthestrategyofexploringchildnodes. Themethodis
believedtoadaptautomaticallytotheeffectivesmoothnessofthetree. Churchilletal.[7]addressed
thecomplicationsassociatedwithsimultaneousanddurativeactionsbyextendingthealpha–beta
search process. However, the large branching factors caused by independently acting objects still
remainedasubstantialchallenge. Methodssuchascombinatorialmulti-armedbanditshaveattempted
toaddressthechallengeoflargebranchingfactors[8,9]. OntañónandBuro[10]addressedverylarge
statespaceandbranchfactorsbycombiningthehierarchicaltasknetwork(HTN)planningapproach
withgametreesearchtodevelopwhatwasdenotedasadversarialHTN(AHTN).Here,ratherthan
exploringtheentirecombinationofpossibleactions,HTNplanningcouldguidethesearchdirection
basedondomainknowledge. AHTNalsoextendedtheHTNplanningapproachtoaccommodate
simultaneousanddurativeactions,andappliedHTNplanningdirectlyintothegamesearchtreeto
takeadvantageofstandardoptimizationssuchasalpha–betapruning.
AlthoughAHTNhasachievedgoodperformancecomparedtoothersearchalgorithms[10],itstill
suffersfromthefollowingweaknesses.
1. TheHTNdescriptionusedbyAHTNcannotexpresscomplexrelationshipsamongtasksand
accommodatetheimpactoftheenvironmentontasks. Forexample,considerataskinvolvingthe
forcedoccupationofafortifiedenemyemplacement,denotedasthecapture-blockhousetask,which
consistsoftwosubtasks. Thefirstsubtaskinvolvesluringtheenemyawayfromtheemplacement
(denotedastheluring-enemysubtask),andthesecondsubtaskinvolvesattackingtheemplacement
(denotedastheattackingsubtask). Here,thecapture-blockhousetaskwillfailiftheattackingsubtask
fails. However,failureoftheluring-enemysubtaskwouldnotnecessarilyleadtoanoverallfailure
ofthecapture-blockhousetask,butwouldrathertendtoincreasethecostofcompletingtheparent
taskbecausetheattackingsubtaskcanstillbeexecutedeventhoughtheluring-enemysubtaskfails.
ThistypeofrelationcannotbeexpressedbytheHTNdescriptionemployedinAHTN.Relations
relatedtoconditionswheresubtasksaretriggeredbytheenvironmentorwheretheexecution
resultsofaparenttaskdependonboththeenvironmentandthesuccessfulcompletionofits
subtasksalsocannotbeexpressedbytheHTNdescriptionemployedinAHTN.
2. AHTN planning cannot effectively address task failures that occur during plan execution.
The planning process in AHTN generates a new plan at each frame of the game whenever
idle units which are not assigned or execute actions exist. The new plan processing cannot
cancelassignedactionsofthepreviousplanning. Ifunitshavebeenassignedactionsinprevious
Appl.Sci.2017,7,872 3of24
planning, they will keep executing those actions until those actions are failed or completed.
InAHTN,eachfailedtaskofthepreviousplanissimplyremovedatthetimeoffailurewithout
consideringitsimpactsonremainingexecutingtasks.Theremainingtaskswillcontinueexecuting
thoughtheseexecutionsmaybemeaningless. Toillustratethispoint,wenotethat,ifthefailure
ofataskleadstothefailureofitsplandirectly,theremainingexecutingtasksoftheplanshould
notbecontinuedexecutionbecausetheycannotaffectthefailureoftheplan. Theremaining
executingtasksshouldthereforebeterminatedsothatthereleasedresourcescanbeemployedto
repairfailedtasksorformulateanewplan. Thus,whenonetaskfailsandcannotberepaired,
allrelatedtasksintheplanshouldbeterminated,andtheAIplayershouldattempttorepairthe
tasktomaintainthevalidityoftheoriginalplan.
In this paper, we propose a modified AHTN algorithm with failed task repair functionality,
denotedasAHTN-R,toaddressthetwoproblemsdiscussedabove. First,AHTN-RemploysanHTN
description extended by adding the elements essential task, phase, and exit condition to enhance its
capability for expressing complex relationships among tasks and for accommodating the impacts
oftheenvironment. Second,AHTN-RemploysamonitoringstrategybasedontheextendedHTN
description to identify all tasks affected by a failed task. Finally, we develop a novel strategy for
repairingfailedtasksbasedonaprioritizedlistofalternativeplans. Theprioritiesofalternativeplans
aregeneratedbysortingallnodesofthegamesearchtreeaccordingtotheirprimaryfeatures,andwe
employ the valid alternative plan with the highest priority to repair the failed task. This method
canreducetimeconsumptionbytakingadvantageofhistoricalinformation. Ineachdecisioncycle,
themethodattemptstorepairfailedtasks,followedbyanewplanningprocess.
The remainder of this paper is organized as follows. Related work is presented in Section 2.
TheextendedHTNdescriptionispresentedinSection3.TheAHTN-Rplanningalgorithmispresented
inSection4. ExperimentalresultsindicativeoftheperformanceoftheproposedAHTN-Ralgorithm
arecomparedwiththeperformancesofotheralgorithmsinSection5. Finally,weconcludethepaper
andpresentanindicationoffutureworkinSection6.
2. RelatedWork
SincethefoundationofHTNplanningwasfirstproposedbySacerdoti[11]in1975,manyHTN
plannershavebeenproposed,suchasthesimplehierarchicalorderedplanner(SHOP)[12]andits
successorSHOP2[13]. CombiningHTNplanningwithsearchmethodsguidedbyhumanknowledge
has been demonstrated to speed up planning dramatically, and this has led some researchers to
attempt to apply this approach in typical video games [14]. Menif et al. [15] applied the Simple
Hierarchical Planning Engine (SHPE) to Steam Box, which is a type of first person shooter game,
usinganalternativeencodingofplanningdata. SoemersandWinands[16]investigatedthereuse
ofHTNplansinvideogames. NumerousexamplesexistofthesuccessfuluseofHTNplannersin
commercialvideogames[17]. However,tothebestofourknowledge,onlyafewattemptshavebeen
madetoapplyHTNplannerstoRTSgames. Muñoz-AvilaandAha[18]employedanHTNplannerto
providehumanplayerswithexplanationsforthereasonsleadingtocurrentstatesoreventsaccording
touserqueries,andforinterrogatingthebehaviorofautonomousplayersundercomputercontrol.
Laagland[19]presentedthedesign,implementation,andevaluationofanHTNplannerinanopen
sourceRTSgamedenotedasSpring. TheplanningoftheRTSgamewasdividedintothreelevelswith
theHTNplannerbeingemployedinthehigheststrategiclevel. Laagland[20]furthersummarizedthe
advantagesanddisadvantagesofHTNplanninginRTSgames. Naveedetal.[21]employedHTN
plannerstoreducethesizeofthepathfindingsearchspaceinRTSgames,andtestedtheiralgorithmin
gamesdevelopedusingORTSin2010. Mostrecentlyin2015,OntañónandBuro[10]developedthe
AHTNalgorithmandappliedittoRTSgames.
Of the above studies, we note that no research other than that of Ontañón and Buro [10] has
consideredtheadversarialfeatureofRTSgames. Moreover,noresearchhasadequatelyaddressedtask
failuresthatoccurduringplanexecutioninRTSgames. However,studiesfocusedinotherfieldshave
Appl.Sci.2017,7,872 4of24
addressedtaskrepairbasedonHTN.Garzónetal.[22]proposedataskrepairstrategybasedonHTN
planningtoadapttofailuresduringtheexecutionoftreatmentplansgeneratedbytherapyplanning
systems. Thestrategyemployedthreemethodsforeffectingtaskrepair: applicationoftheknowledge
base,searchofalternativedecompositions,andoperatornotification. Gateauetal.[23]employedHTN
torepairfailedtaskstoaslocalizedanextentaspossibleinahigh-leveldistributedarchitecturefor
multi-robotcooperation. Thecoreconceptofthestrategywastore-planonlyasub-treeoftheplan
basedonahierarchicalstructure. Ayanetal.[24]designedanextensionofSHOPtorepairfailedtasks
byintroducingatask-dependencygraph. Here,onlythatportionoftheoriginalplanrelatedtothe
failedtaskwassubjectedtore-planning. Thetask-dependencygraphwasemployedtorecordcausal
linksinthetasknetwork,andwasthenappliedtoeliminatetheeffectsofre-planningontheother
portionsoftheoriginalplan,andtoreducethecostofre-planning. Theapproachappliedre-planning
onlytotheaffectedpartsofthetasknetworkusingHTN.However,manyoftheseapproachesfailed
totakeadvantageoftheinformationgeneratedinthepreviousplanningprocesstoreducethecostof
taskrepair.
3. ExtendedHTNDescription
ThissectionpresentsanextendedHTNdescriptiontoexpresscomplexrelationshipsamongtasks
andaccommodatetheimpactsoftheenvironmentontasks. Wefirstanalyzetherequirementsfor
extendingtheHTNdescription,andthenpresentaformaldefinitionoftheextendedHTNdescription.
3.1. RequirementsAnalysis
The HTN planning approach generates plans by decomposing tasks recursively into smaller
subtasks [25]. HTN planning includes two types of tasks: primitive tasks and compound tasks.
Primitivetaskscorrespondtoactionsthatcanbeexecutedbyanagentdirectlyinthegameenvironment,
suchasaspecificmovement. Compoundtasksrepresentgoalsthatmustbeachievedbyahigh-order
plananddecomposedintoatasknetwork.
3.1.1. EssentialTaskAttribute
InAHTNplanning,theresultofacompoundtaskdependsontheresultsandlogicalrelationships
ofitssubtasks. Itstaskdecompositiontreeisanand-or-treewithlogicalrelationshipsbetweensubtasks
thatareeitherANDorOR.TheANDrelationshipindicatesthatthefailureofanysubtaskwillleadtoa
failureofitsparenttask. TheORrelationshipindicatesthatthesuccessofanysubtaskwillleadtothe
successofitsparenttask.However,somedomainknowledgeregardingthecompletionofataskcannot
berepresentedbyanand-or-tree. Returningtothecapture-blockhousetaskexamplediscussedinthe
introduction,wenotethatthesuccessoftheluring-enemysubtaskmaydecreasethecostofcompleting
theparenttask,butitsfailurewouldnotleadtothefailureofitsparenttask. However,thesuccess
andfailureofthecapture-blockhousetaskiscompletelydependentonthesuccessandfailureofthe
attackingsubtask. Thesemorecomplexlogicalrelationshipscannotberepresentedbytheformationof
anand-or-tree.
Torepresenttaskdecompositionreflectingmorecomplexlogicalrelationships,weintroducean
attributedenotedasessentialtask,whosevalueiseithertrueorfalse,andthisattributeisappliedto
alltasksoftheHTN.Here,thefailureofanyessentialtaskwillautomaticallyleadtothefailureofits
parenttaskwithoutconsiderationfortheresultsofanyothersubtasks. Fortwosubtaskst andt ,
1 2
theirlogicalrelationshipsarerepresentedasfollows:
1. iftheessentialtaskattributesofbotharetrue,therelationshipbetweent andt isAND;
1 2
2. iftheessentialtaskattributesofbotharefalse,therelationshipbetweent andt isOR;
1 2
3. iftheessentialtaskattributeofonlyoneistrue,therelationshipisneitherANDnorOR.
Appl.Sci.2017,7,872 5of24
3.1.2. PhaseandExitConditionAttributes
Theresultofacompoundtasknotonlydependsontherelationshipsandresultsofitssubtasks,
butsometimesalsodependsontheenvironment,whichcanchangedynamically. Theseimpactsfrom
theenvironmentcanbeofthefollowingtwotypes.
1. The environment can control the start of tasks once previous tasks have been completed.
For example, the task associated with the sudden concealed attack on an enemy, denoted as
the ambushing-enemy task, consists of two subtasks: the first is associated with movement to
thestagingareaoftheambuscade,denotedasmove-to-ambushing-area,andtheactualactivity
of attacking, denoted as attack-enemy. It is obvious that the attack-enemy subtask could not
be executed immediately upon completing the move-to-ambushing-area subtask because the
attack-enemytaskcannotbeexecuteduntiltheenemyresidesintheambuscadearea.
2. Theenvironmentcouldnecessitatethefailureofatask. Fortheambushing-enemytaskexample,
the move-to-ambushing-area subtask must be completed before the enemy has passed the
ambuscadestagingarea;otherwise,theambushing-enemytaskwouldfail.
WhiletheAHTNalgorithmcanaccommodatethefirsttypeofimpactusingthepreconditionsof
tasks,thesecondtypeofenvironmentalimpactcannotbeaddressed. Inthispaper,weproposetheexit
conditionattributetoaddressthesetwotypesofenvironmentalimpacts. Theexitconditionattributecan
bedividedintosufficientexitconditionandnecessaryexitconditionattributes. Ifataskhassomenecessary
exitconditions,itssubsequenttaskcannotbeexecuteduntilallnecessaryexitconditionsaresatisfied.
Ifataskhassomesufficientexitconditions,itsparenttaskwillfailwhenanysufficientexitconditionis
satisfiedbutthetaskisnotcompleted. TheexitconditionattributeiswrittenusingaLisp-likesyntax
likethepreconditiondefinedin[13].
Thephaseattributeisproposedtodescribetherelationshipsbetweensubtasksthatreflectthefact
thattheexecutionofacompoundtaskcanbedividedintoseveralsequentialsteps,andeachstepmay
includeparallelsubtasks. Becausetasksinthesamephaseareoftenaffectedbythesameexitconditions,
weaddthesetofexitconditionstothephaseratherthanaddingthemtoeachtask.
For an example, we return to the ambushing-enemy task, where we have modified the task to
consistofthreesubtasks: move-to-ambushing-area,surveil-enemy,andattack-enemy. Themodification
includesthesubtasksurveil-enemyreflectingtherequirementtofirstdeterminethattheenemyresides
in the ambuscade area prior to attacking. The first two subtasks are executed in parallel using
differentunitsandtheattack-enemycanbeexecutedonlywhenthefirsttwosubtasksarecompleted.
Wedividetheexecutionofambushing-enemyintotwophases. Thesubtasksmove-to-ambushing-areaand
surveil-enemyareincludedinthefirstphase,whichisdenotedasthepreparatory-phase. Theattack-enemy
subtaskisincludedinthesecondphase,denotedastheattack-phase. Thepreparatory-phasehasboth
sufficient and necessary exit conditions, where the ambushing-enemy task will fail if the subtasks of
the preparatory-phase have not been completed by the point at which the enemy has passed the
ambushing-area (i.e., the sufficient exit condition is satisfied), and the attack-enemy task will not be
initiatedifthesubtasksofthepreparatory-phasehavebeencompletedwhiletheenemyhasnotpassed
theambushing-area(i.e.,thenecessaryexitconditionisnotsatisfied). Thedatastructuresemployedin
theexampleareshowninFigure1,wheretheunitsdenotedas?unitid1and?unitid2willambushthe
enemydenotedas?enemyattheambushing-areadenotedas?place. Inthepreparatory-phase,?unitid1
is allocated to move to ?place and ?unitid2 surveils ?enemy. If the preparatory-phase is completed,
?unitid1willattack?enemy. Additionaldefinitionsareprovidedinthefollowingsubsections.
Appl.Sci.2017,7,872 6of24

Appl. Sci. 2017, 7, 872 6 of 25

FFiigguurree 11.. AAnn eexxaammppllee ooff tthhee aammbbuusshhiinngg--eenneemmyy ttaasskk..
Including the phase attribute during the decomposition of a compound task is the main
Includingthephaseattributeduringthedecompositionofacompoundtaskisthemaindifference
difference between the AHTN used in this paper and that in [10]. The definition of the phase attribute
betweentheAHTNusedinthispaperandthatin[10]. Thedefinitionofthephaseattributeisasfollows.
is as follows.
DDeeffiinniittiioonn 1 (1p.h ase()p.haTseh)e: phaTsehea ttribpuhtaeseis defiantterdibuaste a tuipsl e: d(cid:104)neafimnee,d methaosd , suaf exitctounpsleet:,
〈n(cid:1866)e(cid:1853)ce(cid:1865)x(cid:1857)it,c(cid:1865)on(cid:1857)(cid:1872)sℎet(cid:1867),(cid:1856)t,a(cid:1871)s(cid:1873)k(cid:1858)se(cid:1857)t(cid:1876)(cid:105)(cid:1861),(cid:1872)w(cid:1855)(cid:1867)h(cid:1866)er(cid:1871)e(cid:1857)t(cid:1872)h,(cid:1866)e(cid:1857)fo(cid:1855)l(cid:1857)lo(cid:1876)w(cid:1861)(cid:1872)i(cid:1855)n(cid:1867)g(cid:1866)d(cid:1871)e(cid:1857)fi(cid:1872)n,(cid:1872)it(cid:1853)i(cid:1871)on(cid:1863)s(cid:1871)(cid:1857)a(cid:1872)p〉p, lwy.here the following definitions apply.
•• (cid:1866)n(cid:1853)am(cid:1865)e(cid:1857) isist htheen anmameoef othfe thphea pshe.ase.
•• (cid:1865)m(cid:1857)et(cid:1872)hℎo(cid:1867)d(cid:1856)c ocnosnisstisstosf othfe thnaem neaomfeth oefm tehteh omdetothwohdi ctho twhehpichhas tehbee lpohnagssea bnedloanligsst oafnpda raa mlisett eorsf fpoarrtahmepehteasres.
• fsourf tehxeit pcohnassee.t isthesetofsufficientexitconditions. Eachelementisalogicexpressionofliteralswhich
• (cid:1871)co(cid:1873)n(cid:1858)s(cid:1857)i(cid:1876)st(cid:1861)(cid:1872)o(cid:1855)f(cid:1867)t(cid:1866)he(cid:1871)(cid:1857)n(cid:1872)a mise tahned saetli sotf osfupffaircaiemnett eexrsit. conditions. Each element is a logic expression of literals
• wnehciecxhit ccoonnssiestti osft hthees entaomfnee acensdsa ar yliesxt iotfc ponadraitmioentse.rEs.a chelementisalogicexpressionofliteralswhich
• (cid:1866)co(cid:1857)n(cid:1855)s(cid:1857)i(cid:1876)st(cid:1861)(cid:1872)o(cid:1855)f(cid:1867)(cid:1866)th(cid:1871)e(cid:1857)(cid:1872)n aims ethaen sdeta olifs nteocfespsaarraym eextietr cso.nHdietrioe,nss.u Ebsaecqhu eenletmpheanste iss caa nlobgeice exxecpurteesdsioonnl yofw lhiteenraallsl
wnehceicshsa rcyonexsiitstc oonfd tihtieo nnsaamrees aatnisdfi ead .list of parameters. Here, subsequent phases can be executed
• otanslkys ewthisetnh ealsle nteocfessusbatrays ekxsitth caotnsdhiotuiolndsb aerceo msaptliestfeidedto. accomplishacompoundtask. Thesubtaskscanbe
• (cid:1872)(cid:1853)(cid:1871)(cid:1863)(cid:1871)(cid:1857)(cid:1872) is the set of subtasks that should be completed to accomplish a compound task. The
eithercompoundtasksorprimitivetasks. Eachelementintasksetconsistsofthesubtask’snameandalist
subtasks can be either compound tasks or primitive tasks. Each element in (cid:1872)(cid:1853)(cid:1871)(cid:1863)(cid:1871)(cid:1857)(cid:1872) consists of
ofparameters.
the subtask’s name and a list of parameters.
Because a phase is executed sequentially, any failure of a phase will lead to the failure of the
Because a phase is executed sequentially, any failure of a phase will lead to the failure of the
methodassociatedwiththatphase. Theresultofaphasedependsontwofactors: tasksandsufficientexit
method associated with that phase. The result of a phase depends on two factors: tasks and sufficient
conditions. Accordingly,thefollowingfourconditionsleadtothefailureofaphase:
exit conditions. Accordingly, the following four conditions lead to the failure of a phase:
1. Anyessentialtaskofthephasehasfailed.
1. Any essential task of the phase has failed.
2. Anysufficientexitconditionissatisfied,withoneormoreessentialtasksstillexecuting.
2. Any sufficient exit condition is satisfied, with one or more essential tasks still executing.
3. Anysufficientexitconditionissatisfied,andnosubtaskshavebeencompleted.
3. Any sufficient exit condition is satisfied, and no subtasks have been completed.
44.. AAllll ssuubbttaasskkss hhaavvee ffaaiilleedd..
FFoorr 11,, tthhee ffaaiilluurree ooff aannyy eesssseennttiiaall ttaasskk mmeeaannss tthhaatt tthhee ppaarreenntt ttaasskk hhaass ffaaiilleedd,, aanndd iittss pphhaassee iiss aallssoo
ccoonnssiiddeerreedd aass ffaaiilleedd.. FFoorr2 2,,u unncocommpplelteetdede sessesnetniatliatla staksskcsa cnannontocto ncotinntuineueex eecxuetcinugtinwgh wenhaensu affi scuiefnfitcieexnitt
ecxointd ciotinodnitiisosna itsis sfiaetdis,faiendd, athnedp tahree nptatraesnkt atalssok faalislos. fIanilasd. Idni taiodnd,iaticocno,r dacincogrtdoin3g,t thoe 3p,h tahsee wphiallsefa wiliwll hfaeinl
wnohesunb ntaos ksusbatraesckosm arpel ectoedmapnledteads uafnfidci ean stuefxfiitciceonnt deixtiiotn coisndsaittiiosnfi eisd .saFtiinsafilelyd,.a Ftiansaklloyb, vai otausskly ofbavilisoiufsalyll
fiatsilssu ibf taalsl kitss hsauvbetafsakilse dha.ve failed.
3.2. Definition of the Extended HTN Description
Appl.Sci.2017,7,872 7of24
3.2. DefinitionoftheExtendedHTNDescription
AnHTNproblemcanbedefinedasatuple:(cid:104)S, M, O, TN, γ(cid:105),wherethefollowingdefinitionsapply:
• S is the set of current world states, and consists of all information that is relevant to the
planningprocess.
• Misthesetoftaskdecompositionmethods. Eachmethodm ∈ Mcanbeappliedtodecomposea
taskintoasetofsubtasks.
• Oisthesetofoperators. Eachoperatoro ∈Oisanexecutionofaprimitivetask.
• TNisthecurrenttasknetwork. Itisatreewhosenodesaretasks,methods,orphases.
• γisthestatetransformfunction. Givens ∈ S,γ(s,o)definesthetransitionofthestatewhenan
primitivetaskisexecutedbyanagent. Ifγ(s,o),theoperatoroisnotapplicableins.
The definition of a method in the proposed extended description is similar to the definition
employedinpreviousHTNdescriptions. However,ourmethodusesthelistofphasesratherthansets
ofsubtasks,andaddstheattributedenotedasessential.
Definition2(method). Amethodisdefinedasatuple: (cid:104)name, task, precon, phases, essential(cid:105),wherethe
followingdefinitionsapply.
• nameisthenameofthemethod.
• taskisthenameofthetasktowhichthemethodisapplied.
• preconisthelogicalpreconditionsforthemethod,whichshouldbesatisfiedwhenthemethodisapplied.
• phasesisthelistofphases. Thesequenceoftheelementsin phasesistheexecutionsequence.
• essentialisaBooleanvalue,andistruetoindicatethatatasktowhichthemethodisappliedisanessentialtask.
ThedefinitionofanoperatorisalsosimilarwiththatinpreviousHTNdescriptionsexceptfor
addingtheessentialattribute.
Definition3(operator). Theoperatorisdefinedasatuple: (cid:104)head, cons, dels, adds, essential(cid:105),wherethe
followingdefinitionsapply.
• headistheprimitivetaskthatcanbeappliedbythisoperator.
• consisthepreconditionthatmustbesatisfiedbeforetaskexecution.
• delsarethedeleteeffects.
• addsaretheaddeffects.
• essentialisaBooleanvalue,andistruetoindicatethatthetasktowhichtheoperatorisappliedisanessentialtask.
4. AHTN-RPlanningAlgorithm
TheoverallframeworkoftheAHTN-Ralgorithmisprovidedfirstly,andthefunctionsofitsdifferent
componentsareexplained.Then,themonitoringstrategyemployingtheextendedHTNdescriptionis
proposedtoaddresstheissueoffailedtasks.Finally,wediscusstheproposedfailedtaskrepairstrategy.
4.1. ATHNRFramework
The overall framework of the AHTN-R algorithm is illustrated in Figure 2. The process flow
consists of three main components: plan generation, plan execution, and task repair. The plan
generation component provides the best plan to the plan execution component, and provides a
prioritized list of alternative plans to the task repair component. The plan execution component
processesthefailedtasksresponsibleforplanfailureandsendsthetaskneedingrepairtothetask
repaircomponent. Thetaskrepaircomponentattemptstorepairfailedtasksatthebeginningofeach
decisioncycle.
Appl. Sci. 2017, 7, 872 8 of 25
4.1. ATHNR Framework
The overall framework of the AHTN-R algorithm is illustrated in Figure 2. The process flow
consists of three main components: plan generation, plan execution, and task repair. The plan
generation component provides the best plan to the plan execution component, and provides a
prioritized list of alternative plans to the task repair component. The plan execution component
processes the failed tasks responsible for plan failure and sends the task needing repair to the task
repair component. The task repair component attempts to repair failed tasks at the beginning of each
Appl.Sci.2017,7,872 8of24
decision cycle.

Figure 2. Overall framework of the adversarial hierarchical task network (AHTN)-R algorithm.
Figure2.Overallframeworkoftheadversarialhierarchicaltasknetwork(AHTN)-Ralgorithm.
At each decision cycle, when there are idle units, the plan generation component will search the
best plan using the AHTN algorithm with the extended HTN description. This paper modified the
Ateachdecisioncycle,whenthereareidleunits,theplangenerationcomponentwillsearchthe
part of the AHTN algorithm obtaining the next action. In AHTN, the subtasks can be obtained if a
bestplanusimnegthtohde isA aHppTlieNd. aHlogwoervitehr, mwitwh tihthe itnhtreodeuxctteionnd oef dphaHseT inN AdHeTsNc-rRi,p thtieo pnh.asTehs wisilpl baep oebrtaminoedd ifiedthepart
oftheAHTNratahlegr othrainth thme suobbttaaskins iifn ag mtehtheodn ies xatppalcietdio. Tnh.e IAnHATNH-RT aNlg,otrhithems uacbcetassseks sthce agnenebreatoedb ptahainsees difamethod
to obtain the next primitive task. Algorithm 1 shows the method employed to obtain the next
isapplied.However,withtheintroductionofphaseinAHTN-R,thephaseswillbeobtainedratherthan
executable action based on the HTN plan N and execution pointer t, which keeps track of which parts
thesubtasksofi fNa hmavee tahlroedadyis baepenp elxieecdu.teTdh. TehAe rHesTulNt o-fR Aalglogroitrhimth 1m is aac pcreimssiteivset htaeskg oern aenr eamtepdty pseht,a sestoobtain
thenextpriminditiicvateintga asnk .abAselngcoe roift hanm av1aislahbolew prsimthiteivem taestkh. oLdineesm 1–p4 lsohyowed thtaot, oifb ntoa tiansktsh heavnee yxett ebxeeenc utableaction
executed, the algorithm returns the first primitive task in N. Lines 7–19 show that, if all tasks in the
basedontheHTNplanNandexecutionpointert,whichkeepstrackofwhichpartsofNhavealready
same phase with t have been executed, the algorithm searches the first primitive task from the next
beenexecutpehda.seT. Ihf enor epsriumlittivoef tAasklgs oarrei tfohumnd 1ini sthae nperxit mtasikti, vane etmaspkty osert aisn reetumrnpedty. Lsineets, 2in0–d31ic sahtoiwn ganabsence
ofanavailabthlaet pthrei amlgiotriivthemt saesakrc.hLesi nfoer sa p1r–im4istihveo twaskt hfroamt, tihfen reomtaainsiknsg thaaskvse iny theet sbaemeen pheaxsee cthuatte hdav,et healgorithm
not been executed.
returnsthefirstprimitivetaskinN.Lines7–19showthat,ifalltasksinthesamephasewiththave
beenexecute d,thealgorithmsearches thefirstprimitivetaskfromthenextphase. Ifnoprimitivetasks
arefoundinthenexttask,anemptysetisreturned. Lines20–31showthatthealgorithmsearchesfora
primitivetaskfromtheremainingtasksinthesamephasethathavenotbeenexecuted.
Algorithm1ReturnsnextAction(N,t)
1. Ift=⊥then
2. GettherootofN,denotedastherootTask
3. ReturnnextAction(N,rootTask)
4. EndIf
5. Acquirethephaseptowhichtbelongs
6. Getalltasksdenotedassubtasksofp
7. Ifalltasksinsubtaskshavebeenexecutedthen
8. Getthenextphasep(cid:48)ofp
9. Ifp(cid:48)=⊥then
10. Return(cid:11)
11. EndIf
12. Getalltasksdenotedassubtasks(cid:48)ofp(cid:48)
13. Forallt(cid:48)∈subtasks(cid:48)do
14. IfnextAction(N,t(cid:48))(cid:54)=(cid:11)then
15. ReturnnextAction(N,t(cid:48))
16. EndIf
17. EndFor
18. Return(cid:11)
19. EndIf
20. Forallt(cid:48)(cid:48) ∈subtasksthathavenotbeenexecuteddo
21. Ift(cid:48)isaprimitivetaskthen
22. Returnt(cid:48)(cid:48)
23. EndIf
24. Getthefirstphasep(cid:48)(cid:48) ofthemethodappliedtot(cid:48)(cid:48)
25. Getalltasksdenotedassubtasks(cid:48)(cid:48) ofp(cid:48)(cid:48)
26. Forallt(cid:48)(cid:48)(cid:48) ∈subtasks(cid:48)(cid:48) do
27. IfnextAction(N,t(cid:48)(cid:48)(cid:48))(cid:54)=(cid:11)then
28. ReturnnextAction(N,t(cid:48)(cid:48)(cid:48))
29. EndIf
30. EndFor
31. EndFor
32. Return(cid:11)
Appl.Sci.2017,7,872 9of24
Once the best plan is generated, the AHTN-R will obtain the prioritized listing of alternative
plansgeneratedbyAHTNthroughsortingthenodesofthegamesearchtree. Themethodofsorting
plansisdiscussedinSection4.3. Duringplanexecution,thetaskmayfailbecauseofthedynamics
oftheRTSgameenvironment. Onceataskfails,AHTN-Rwillprocessthefailedtaskaccordingto
themonitoringstrategydiscussedinSection4.2,whichseekstoidentifyandterminatethesmallest
possiblesetoftasksaffectedbythefailedtask. Inthisway,theeffectsofafailedtaskcanbelimitedto
thesmallestscopepossible.
At the beginning of each decision cycle, AHTN-R first attempts to repair the failed task by
selectingasuitableplanfromthelistofalternativeplansaccordingtotheirpriorities. AHTN-Rwill
thenexecutethetasksofthealternativeplancorrespondingtothetasksaffectedbythefailedtask.
IfAHTN-Risunabletofindasuitablealternativeplan,itwillgenerateanaltogethernewplanforthe
failedtaskbyAHTN.ThisprocessisdiscussedattheendofSection4.3.
4.2. FailedTaskMonitoringStrategyUsingtheExtendedHTNDescription
Onceataskfails,anAIplayermustidentifyandterminatealltasksaffectedbythefailedtask.
Toaccomplishthis,thesourcesoftheplanfailureareanalyzedfirst. Planfailurecanbeattributed
totwosources: thefailureofactionsandthestateofthesufficientexitconditionsattribute. Anaction
willfailwhenitsunitisdestroyedorwhenavailableresourcescannotbeobtainedtofulfilltheaction.
Aplanmayalsofailifthesufficientexitconditionissatisfied,asdiscussedinSection3.1.2. Taskfailure
triggersthemonitoringstrategy. Algorithm2showsthemainprocessofthestrategyemployedto
monitorfailedtasks.
TheinputofAlgorithm2isthefailedtaskt,andtheresultofAlgorithm2isalistdenotedas
RepairTaskList,whichcontainsthetasksrequiringrepair. Lines1–4showthedifferentstrategiesfor
addressingessentialandunessentialtasks. Lines5–15showhowfailedcompoundtasksareaddressed.
Lines7–13showthat,ifafailedcompoundtaskhasavailablemethods,itwillbesavedandrepairedin
thenextdecisioncycle. Onlywhenataskhasnoavailablemethodswillitbeconsideredasthefailed
task. Inthisway,thenumberofaffectedtaskswillbeassmallaspossible,whichwillreducetheextent
ofrepairrequired. Line15showsthestrategyforaddressingprimitiveandcompoundtaskswithout
availablemethods.
Algorithm2ReturnTaskFailed(t)
1. IfEssential(t)= falsethen
2. SubTaskFailed(t)
3. Return
4. EndIf
5. Iftisacompoundtaskthen
6. Acquireallmethodsmoft
7. Foreachmethodm∈ M∧status(m)!=failed
8. GetthepreconditionsetCofm
9. Ifallc∈Cissatisfiedthen
10. AddtintoRepairTaskList
11. Return
12. Endif
13. EndFor
14. Endif
15. SubTaskFailed(t)
According to the extended HTN description, each task, except the root node task, belongs
to a single phase, and whether the phase corresponding to a failed task has also failed must be
determined. Algorithm3showsthestrategyforaddressingthecorrespondingphaseaccordingto
theHTNdescriptionpresentedinSection3.2. TheinputofAlgorithm3isataskt. Line2showsthat,
Appl.Sci.2017,7,872 10of24
ifthefailedtaskistherootnodetask,itneednotberepaired. Ifthephaseisfailed,Algorithm4and
Algorithm5arecalledtoterminatetheaffectedtasksandprocesscompoundtasks. Thefailedphase
servesastheinputofAlgorithms4and5. Algorithm4showsthestrategyforprocessingthefailed
phase. Ifthetaskbelongingtothefailedphaseisacompoundtask,itssubtaskswillbeprocessed,and,
ifitisaprimitivetask,itwillbeterminateddirectly. Algorithm5showsthestrategyforprocessingthe
methodtowhichthefailedphasebelongs. Lines2–6showthatallunfinishedphasesofthemethod
arefailedbecausetheyaresequential.
Algorithm3SubTaskFailed(t)
1. Acquirethephasepcorrespondingtot
2. Ifp=⊥thenreturn
3. EndIf
4. Ifoneofthesufficientconditionsofpismetthen
5. Ifanyessentialtasksofparefailedthen
6. PhaseFailed(p);
7. MethodFailed(p);
8. ElseIfphasnoessentialtask
9. ∧(alltasksofpwereexecutedordidnotstart)
10. PhaseFailed(p);
11. MethodFailed(p);
12. EndIf
13. EndIf
14. ElseIfanyessentialtasksofpfail∧
15. (phasnoessentialtask∧alltasksofpfail)
16. PhaseFailed(p);
17. MethodFailed(p);
18. EndIf
19. EndIf
Ifanysufficientexitconditionissatisfied,theAIplayerwilldeterminewhetherthephasetowhich
the sufficient exit condition belongs has failed. The processing involved is similar to that given by
Algorithm3exceptthatitsinputparameterisasufficientexitconditionratherthanafailedtask.
Algorithm4PhaseFailed(p)
1. Getalltasksdenotedassubtasksofp
2. Foralltaskst∈subtasks
3. Ifstatus(t)= finished∨status(t)= failedthen
4. continue;
5. EndIf
6. Iftisacompoundtaskthen
7. Getthemethodmappliedtot
8. GetallphaseswithinPhasesofm
9. Forallphasesq∈Phases
10. PhaseFailed(q)
11. EndFor
12. Else
13. Cancel(t)
14. EndIf
15. EndFor