Table Of Content

Studies in Systems, Decision and Control 379 Vassil Sgurev Vladimir Jotsov Janusz Kacprzyk Editors Advances in Intelligent Systems Research and Innovation Studies in Systems, Decision and Control Volume 379 SeriesEditor JanuszKacprzyk,SystemsResearchInstitute,PolishAcademyofSciences, Warsaw,Poland The series “Studies in Systems, Decision and Control” (SSDC) covers both new developments and advances, as well as the state of the art, in the various areas of broadly perceived systems, decision making and control–quickly, up to date and withahighquality.Theintentistocoverthetheory,applications,andperspectives on the state of the art and future developments relevant to systems, decision making,control,complexprocessesandrelatedareas,asembeddedinthefieldsof engineering,computerscience,physics,economics,socialandlifesciences,aswell astheparadigmsandmethodologiesbehindthem.Theseriescontainsmonographs, textbooks, lecture notes and edited volumes in systems, decision making and control spanning the areas of Cyber-Physical Systems, Autonomous Systems, Sensor Networks, Control Systems, Energy Systems, Automotive Systems, Biological Systems, Vehicular Networking and Connected Vehicles, Aerospace Systems, Automation, Manufacturing, Smart Grids, Nonlinear Systems, Power Systems, Robotics, Social Systems, Economic Systems and other. Of particular valuetoboththecontributorsandthereadershiparetheshortpublicationtimeframe and the world-wide distribution and exposure which enable both a wide and rapid disseminationofresearchoutput. IndexedbySCOPUS,DBLP,WTIFrankfurteG,zbMATH,SCImago. AllbookspublishedintheseriesaresubmittedforconsiderationinWebofScience. Moreinformationaboutthisseriesathttp://www.springer.com/series/13304 · · Vassil Sgurev Vladimir Jotsov Janusz Kacprzyk Editors Advances in Intelligent Systems Research and Innovation Editors VassilSgurev VladimirJotsov InstituteofInformation UniversityofLibraryStudies andCommunicationTechnologies andInformationTechnologies BulgarianAcademyofSciences Sofia,Bulgaria Sofia,Bulgaria JanuszKacprzyk SystemsResearchInstitute PolishAcademyofSciences Warsaw,Poland ISSN2198-4182 ISSN2198-4190 (electronic) StudiesinSystems,DecisionandControl ISBN978-3-030-78123-1 ISBN978-3-030-78124-8 (eBook) https://doi.org/10.1007/978-3-030-78124-8 ©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNature SwitzerlandAG2022 Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether thewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuse ofillustrations,recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,and transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar ordissimilarmethodologynowknownorhereafterdeveloped. Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant protectivelawsandregulationsandthereforefreeforgeneraluse. Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinformationinthisbook arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional claimsinpublishedmapsandinstitutionalaffiliations. ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland Contents Balancing Exploration and Exploitation in Forward Model Learning ......................................................... 1 AlexanderDockhornandRudolfKruse UniversalAdversarialPerturbationGeneratedbyUsingAttention Information ....................................................... 21 ZifeiWang,XiaolinHuang,JieYang,andNikolaKasabov FrameworkandDevelopmentProcessforIoTDataGathering ......... 41 MikaSaari,PetriRantanen,SamiHyrynsalmi,andDavidHästbacka AutomatedEnvironmentalMappingwithBehaviorTrees ............. 61 FranciscoGarciaRosas,FrankHoeller,andFrankE.Schneider Multi-objectiveAutomaticClusteringwithGeneRearrangement andClusterMerging ............................................... 87 HongchunQu,LiYin,andXiaomingTang Assessment of Deep Learning Models for Human Activity RecognitiononMulti-variateTimeSeriesDataandNon-targeted AdversarialAttack ................................................ 129 MahbubaTasmin, SharifUddinRuman, TaoseefIshtiak, Arif-ur-RahmanChowdhurySuhan, RedwanHasif, ShahnawazZulminan,andRashedurM.Rahman ComplexMultivaluedHierarchicalLogic(HS-Logic) ................. 161 VassilSgurev Design Considerations for a Multiple Smart-Device-Based PersonalizedLearningEnvironment ................................. 173 MasashiKatsumata IntelligentAssistiveSensorsandSmartSystemsfortheControl andAnalysisofDriverReactionTimes .............................. 185 DavidSanders, MalikHaddad, GilesTewkesbury, TomBarker, MartinLangner,andAlexGegov v vi Contents ImplementationandExperimentalValidationoftheIEC63047 StandardforDataTransferinRadiologicalandNuclearRobotic Applications ...................................................... 205 FrankE.Schneider,DennisWildermuth, FranciscoGarciaRosas, andJanPaepen Data Science Modeling and Constraint-Based Data Selection forEEGSignalsDenoisingUsingWaveletTransforms ................ 241 MagdalenaGarvanova,IvanGarvanov,andVladimirJotsov IntuitionisticFuzziness,StandardandExtendedModality ............ 269 KrassimirAtanassov IntroductiontoOctopus-InspiredSoftRobots:Pipe-Climbing RobotTAOYAKA-SIIandLadder-ClimbingRobotMAMEYAKA ..... 287 KazuyukiIto,YoshihiroHomma,HiroakiShimizu,andYuseiSakuhara AutomaticLogAnalysistoPreventCyberAttacks .................... 315 AndreBrandaoandPetiaGeorgieva GeneralizedNetModelforCollecting,EvaluatingandIncluding ofFactsintheEducationalContent ................................. 341 KrassimirAtanassov, AnthonyShannon, EvdokiaSotirova, ValentinVasilev,andSotirSotirov VirtualizationofThingsinaSmartAgricultureSpace ................ 349 StanimirStoyanov, TodorkaGlushkova, IvanPopchev, andLyubkaDoukovska TurbineHillChartGenerationUsingArtificialNeuralNetwork ....... 369 ArlindoRodriguesGalvãoFilho, FilipedeS.L.Ribeiro, RafaelVianadeCarvalho,andClarimarJoséCoelho Stimuli-BasedControlofNegativeEmotionsinaDigitalLearning Environment ...................................................... 385 RossitzaKaltenborn,MinchoHadjiski,andStefanKoynov IntelligentNetwork-FlowSolutionswithRisksatTransportation ofProducts ....................................................... 417 VassilSgurev,LyubkaDoukovska,andStanislavDrangajov EmbeddedIntelligenceinaSystemforAutomaticTestGeneration forSmoothlyDigitalTransformationinHigherEducation ............. 441 PepaPetrova,IvaKostadinova,andMajidH.Alsulami Simulation Algorithms to Assess the Impact of Aging ontheReliabilityofStandbySystemswithSwitchingFailures ......... 463 KirilTenekedjiev, NataliaNikolova, GuixinFan, MarkSymes, andOanhNguyen Balancing Exploration and Exploitation in Forward Model Learning AlexanderDockhornandRudolfKruse Abstract Forwardmodellearningalgorithmsenabletheapplicationofsimulation- based search methods in environments for which the forward model is unknown. Multiplestudieshaveshowngreatperformanceingame-relatedandmotioncontrol applications.Inthese,forwardmodellearningagentsoftenrequiredlesstrainingtime whileachievingasimilarperformancethanstate-of-the-artreinforcementlearning methods.However,severalproblemscanemergewhenreplacingtheenvironment’s truemodelwithalearnedapproximation.Whilethetrueforwardmodelallowsthe accurate prediction of future time-steps, a learned forward model may always be inaccurateinitsprediction.Theseinaccuraciesbecomeproblematicwhenplanning long action sequences since the confidence in predicted time-steps reduces with increasingdepthofthesimulation.Inthiswork,weexploremethodsforbalancing riskandrewardindecision-makingusinginaccurateforwardmodels.Therefore,we proposemethodsformeasuringthevarianceofaforwardmodelandtheconfidence inthepredictedoutcomeofplannedactionsequences.Basedonthesemetrics,we definemethodsforlearningandusingforwardmodelsunderconsiderationoftheir currentpredictionaccuracy.Proposedmethodshavebeentestedinvariousmotion controltasksoftheOpenAIGymframework.Resultsshowthattheinformationon the model’s accuracy can be used to increase the efficiency of the agent’s training andtheagent’sperformanceduringevaluation. B A.Dockhorn( ) QueenMaryUniversity,London,UK e-mail:[email protected] R.Kruse OttovonGuerickeUniversity,Magdeburg,Germany e-mail:[email protected] ©TheAuthor(s),underexclusivelicensetoSpringerNatureSwitzerlandAG2022 1 V.Sgurevetal.(eds.),AdvancesinIntelligentSystemsResearchandInnovation, StudiesinSystems,DecisionandControl379, https://doi.org/10.1007/978-3-030-78124-8_1 2 A.DockhornandR.Kruse 1 Introduction Forward Model Learning describes the process of learning a model of a pri- ori unknown environments. This enables an agent to anticipate the outcome of plannedactionsequencesandusesimulation-basedsearchtechniquestooptimizeits behavior. Thischapterbuildsuponourrecentworkonforwardmodellearningingames[1] and directly extends our work on forward model learning for motion control tasks[12].Previousstudieshaveshownthatreliableforwardmodelscanbelearned byobservation.Sincenoinformationontheenvironmentisavailable,wehavecho- sen to implement an uninformed training process in which the agent uses random actions to explore its environment. This results in many interactions being wasted dueto(1)repeatingaknowninteractionand(2)neitherfocusingonimprovingthe model nor improving the agent’s performance. As a result, we have observed that the model may be unreliable since large amounts of the state space remain unob- served.Inreturn,theagent’sperformanceduringtheevaluation willbelimitedby thepredictionaccuracyofthetrainedforwardmodel. Motivatedbytheseshortcomings,westudymethodsforimprovingtheefficiency oftheagent’strainingprocess.Therefore,weformulatetwolearninggoals(1)the explorationoftheenvironmenttotrainareliableforwardmodel,and(2)theexploita- tionofpromisingactionsequencestobecomemoreproficientinthegiventask.The maincontributionofthisworkcanbesummarizedby: • TaxonomyofLearningAlgorithms:Wepresentaunifyingviewonreinforcement learning,search,andforwardmodellearningalgorithmsinthecontextoftheagent- environmentinterface. • DecomposedDifferentialForwardModel:Wereviewthedefinitionsofprevi- ouslyproposedmodelbuildingheuristicsandtheirgeneralizationinformofthe decomposedforwardmodel.Asaresult,weproposethedecomposeddifferential forwardmodelwhichwillbeusedthroughoutthisstudy. • Risk Awareness: We propose several methods for measuring the agent’s confi- denceinthelearnedforwardmodel’spredictions.Furthermore,weproposeseveral optimizationgoalstobalanceexplorationandexploitationduringtheagent’strainingprocess. The remainder of this chapter is structured as follows: In Sect.2, we compare theconceptsofreinforcementlearning,search-basedalgorithms,andforwardmodel learningfordecision-making. The followingsections (Sects.3and4)focus onthe introductionofforwardmodellearningandtypesofforwardmodelrepresentations. Section5exemplarilyshowstheusageofGaussianprocessregressionandensemble regressionmodelssuchasarandomforestregression.Forboth,weproposemethods for incorporating the variance of made predictions in the agent’s decision-making process (Sect.5.2). We evaluate proposed methods based on their resultingperfor- manceinthreesimplemotioncontroltasksinSect.6.Foramoredetailedanalysis,we comparetheimpactofproposedmeasuresontheagent’strainingprocessinSect.7. WeconcludeourresultsinSect.8anddiscussopportunitiesforfurtherstudies. BalancingExplorationandExploitationinForwardModelLearning 3 2 TaxonomyofLearningAlgorithms Theagent-environmentinterfacerepresentsageneraldescriptionofalearningsce- nario.Here,theagentisincontinuousinteractionwithanenvironment.Eachofthe agent’sexecutedactionshasthepotentialtoupdatethestateoftheenvironment.In return,theagentcanbeofferedanumericalrewardthatistypicallyassociatedwith thegiventask. Inreinforcementlearning,theagentfocusesonpickingactionstomaximizeits expected reward over time. Given observations of all its previous interactions, the agent tries to estimate the expected value of state-action pairs. Algorithms such asTemporalDifferenceLearning(TDL)[32],Q-learning[35],andtheMonteCarlo (MC)method[30],updatetheexpectedvalueafterarewardsignalhasbeenobserved oraftertheresultofanepisodehasbeenobservedrespectively.Thesetechniquesare calledmodel-freesincetheydonottrytobuildamodeloftheirenvironment. Incontrasttothesesimplerreinforcementlearningalgorithms,whicharestoring thevalueofeachstate-actionpair,methodsindeepreinforcementlearninguseaneu- ralnetworktoapproximatethevaluebasedontheinput,thusdrasticallydecreasing thestorageusedforthemodel[37].Whiledeepreinforcementlearningalgorithms have resulted in an impressive performance in the context of many game-related benchmarks,thenumberofrequiredtrainingexamplestofitallthenetwork’sparam- etersisoftenhighandthusrequiremuchtrainingtime. Simulation-basedsearchalgorithms,suchasminimax[23]orMonteCarlotree search(MCTS)[5],utilizetheenvironment’sforwardmodeltosimulatetheoutcome ofplannedactionsequences.Eachactionsequencerepresentsacandidatesolution. Itssimulationiscalledarolloutandtheresultofsuchasimulationcanbeusedto estimatethevalueofactionsinthesimulatedsequence.Incontrasttoreinforcement learning algorithms, simulation-based search algorithms estimate the value of an action at run-time. Therefore, these algorithms require knowledge of the current state and the game’s model, and return the action with the highest expected value regardingthecurrentstate.Inreturn,theycanbeappliedwithoutpriortraining.Due totheircapabilityofbeingappliedwithouttraining,searchalgorithmssuchasMCTS andOpenLoopSearch[26]haveperformedwellingeneralgame-playingtasks[27]. Dynamicprogrammingalgorithms[21]requireknowledgeoftheenvironment’s forwardmodeltocalculatethetruevalueofanystate-actionpairs.Thisoptimization process can yield a perfect policy for small state and action spaces. However, the highbreadthordepthofthesearchtreeoftenrendersthismethodinfeasible. Next to traditional search schemes, the rolling horizon evolutionary algorithm (RHEA) [15, 16], uses mutation and crossover to optimize the agent’s action sequence. A heuristic value of simulated action sequences can be used as a fit- nessmeasuretoguideevolutionaryoptimization.Thedesignofsuchaheuristiccan drasticallyimpact,theresultingbehavior. In the following, we compare the reviewed methods based on their knowledge of the environment. Given the agent’s action and the current state, the agent will observethenextstateandareward.Bothupdatescanbeencodedinaseparatemodel,