Deep Statistical Comparison for Meta-heuristic Stochastic Optimization Algorithms

141 Pages·2022·3.136 MB·English

by  Tome Eftimov, Peter Korošec

Natural Computing Series
FoundingEditor
GrzegorzRozenberg
SeriesEditors
ThomasBäck ,NaturalComputingGroup–LIACS,LeidenUniversity,Leiden,
TheNetherlands
LilaKari,SchoolofComputerScience,UniversityofWaterloo,Waterloo,ON,
Canada
SusanStepney,DepartmentofComputerScience,UniversityofYork,York,UK
Scope
NaturalComputingisoneofthemostexcitingdevelopmentsincomputerscience,
andthereisagrowingconsensusthatitwillbecomeamajorfieldinthiscentury.This
seriesincludesmonographs,textbooks,andstate-of-the-artcollectionscoveringthe
wholespectrumofNaturalComputingandrangingfromtheorytoapplications.
Moreinformationaboutthisseriesathttps://link.springer.com/bookseries/4190
·
Tome Eftimov Peter Korošec
Deep Statistical Comparison
for Meta-heuristic Stochastic
Optimization Algorithms
TomeEftimov PeterKorošec
ComputerSystemsDepartment ComputerSystemsDepartment
JožefStefanInstitute JožefStefanInstitute
Ljubljana,Slovenia Ljubljana,Slovenia
ISSN 1619-7127
NaturalComputingSeries
ISBN 978-3-030-96916-5 ISBN 978-3-030-96917-2 (eBook)
https://doi.org/10.1007/978-3-030-96917-2
©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNature
SwitzerlandAG2022
Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether
thewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuse
ofillustrations,recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,and
transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar
ordissimilarmethodologynowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Tomymother,father,brother,both
grandmothersandbothgrandfathersfor
givingmetheirunconditional support.
and
Fortheendlessdiscussionsabout nothing
andeverything…
—Tome
TothefamilyIgrewupwith:motherJožica,
fatherLado,andbrotherBorut
and
Tothefamilygrowingupwithme:mybeloved
Tina,andourchildren,Tim,Nik,andNeli
—Peter
Foreword
Optimization is finding the best solution from among all the feasible solutions. In
mathematicsandcomputerscience,thischallengingtaskisbeingtackledbymany
researchersaroundtheworldwhoareinventingvariousapproachesandalgorithms.In
thepast20years,theinventingofnewoptimizationalgorithmshasbecomesomething
of a competition, not only with oneself, but also with other researchers. In theory,
competitive activities have many benefits and help researchers develop important
skills,likeperseveranceandresilience,aswellasprotectingsciencefromstagnation.
However,toavoidunfaircompetition,clearrulesneedtobedefinedandagreedon
whencomparingtheapproachesandalgorithms.TomeEftimovandPeterKorošec
arethatrarebreedofcomputerscientistthatrecognizetheneedtodefinetherulesfor
benchmarkingalgorithmsinaspecificfieldofmeta-heuristicstochasticoptimisation.
Theyprovideafascinatingcompendiumoftheoreticalandpracticalguidesandevena
web-basedtoolformakingastatisticalcomparisonofbothsingle-objectiveandmulti-
objectiveoptimisationalgorithmsinafairway.Thisbookprovidesapathforward,
notonlyinthefieldofmeta-heuristicstochasticoptimisation,butalsoinotherfields
of computer science where approaches and algorithms should be benchmarked in
thecorrectwaybeforeannouncingthemasthebestsolutions.Asaconsequence,the
researcher who wins the competition will succeed as a true scientist, which is the
mostbeautifulfeelingofall.
Ljubljana,Slovenia BarbaraKoroušic´ Seljak
November2021
vii
Preface
Everythingstartedin2015.I(TomeEftimov)camebackfromaconference,andmy
Ph.D.supervisor(BarbaraKoroušic´ Seljak)toldmethatitwouldbenicetomakea
presentation about how many runs we should make toobtain arepresentative data
sampleforastatisticalanalysisofmeta-heuristics.Forthis,Ineededtocollaborate
withmycolleaguePeterKorošec.Theideawastotransferthestatisticalknowledge
appliedinotherdomainstooptimization,wheretheinputwastheexperimentaldata
from running single-objective meta-heuristics. After a few discussions about the
statisticalmethods,webothrealizedthatthiscouldnotbeappliedtoourapplication
scenario.Peterexplainedtomeabouttheweaknesseswhenstatisticalcomparisons
aremadeusingmeta-heuristicsdata.Afterafewhours,IhadimplementedtheDeep
Statistical Comparison approach. The next day, after explaining this to Peter and
discussing the results, he told me that we should publish them in a journal. This
is how the Deep Statistical Comparison story began, with neither of us realizing
atthetimethebenefitswewouldderivefrommakingstatisticalcomparisonsmore
robust.Sincethen,therehavebeenmanylongdiscussions,whichhavefinallyledto
this book. And despite some of the discussions becoming heated, we are happy to
considerourselvesasfriends,notjustcolleagues.
Our initial study about how many optimization runs we would need to obtain
a representative data sample for a statistical analysis of meta-heuristics remains
unfinished. But our diversion into the Deep Statistical Comparison approach was
worthit.
BookContent
Thisbookexplainswhatisrequiredtomakeamorerobuststatisticalcomparisonof
theperformanceachievedbymeta-heuristics.Itdoesnotexplainhowtodevelopa
newsingle-ormulti-objectivemeta-heuristic,orhowtorunitonaspecificoraset
ofprobleminstancesandcollecttheexperimentaldata.Itdealsinwhichstatistical
analysis should be made once the experimental data is collected to obtain robust
statisticaloutcomes.
ix
x Preface
IntendedAudience
Thebookhasbeenwrittenwiththreeaudiencesinmind:
(cid:129)
Studentsinthefieldofmeta-heuristicstochasticoptimization.
(cid:129)
Experiencedresearchersinthefieldofmeta-heuristicstochasticoptimization.
(cid:129)
Engineers who need to select an appropriate optimization algorithm for their
industrialtaskbasedonitsperformance.
ExpectedBackgroundKnowledge
Fewassumptionsaremaderelatingtothebackgroundofthereader.However,they
shouldknowwheretofindmeta-heuristicstochasticoptimizationalgorithms,howto
runthem,thebasicsofhowtheywork,andhowtheexperimentaldataarecollected.
Allthisknowledgeisnotessential,butitwillhelpinunderstandingthelogicbehind
thestatisticalanalysispresentedinthisbook.
BookOutline
Thebookisorganizedforbothnewandexperiencedreaders.Forthenewreadersit
providesthebasicsinoptimizationandstatisticalanalysis,allowingthemtobecome
familiarwiththematerialbeforetheintroductionoftheDeepStatisticalComparison
approach.Experiencedresearcherscanmoverapidlytotheintroductionofthenew
statisticalapproaches.Toaccommodatethereader,thecontentisorganizedintothree
parts:
(cid:129)
Part I: Introduction to optimization, benchmarking, and statistical analysis—
Chaps.2–4.
(cid:129)
Part II: Deep Statistical Comparison of meta-heuristic stochastic optimization
algorithms—Chaps.5–7.
(cid:129)
Part III: Implementation and application of the Deep Statistical Comparison—
Chap.8.
Theaimofthefirstpartistoprovidethebasicsinoptimization,benchmarking,and
statisticalanalysis.Itbeginsbyexplainingtheoptimizationproblem,togetherwith
differentclassificationsthatexistconcerningdifferentcriteriasuchascombinatorial
versusnumerical(basedonthetypeofvariablesusedtodescribetheoptimization
problem)orsingle-versusmulti-objective(basedonthenumberofobjectivesthat
areoptimized).Next,thebenchmarkingprocessisexplainedinmoredetail,focusing
on the four main steps that should be taken with great care when a benchmarking
studyisperformed:(i)identifyingthebenchmarkingobjective,(ii)definingtheopti-
mizationdomain(i.e.,selectionofthealgorithmsandproblems),(iii)definingafair
Preface xi
experimentaldesign,and(iv)statisticallyanalyzingtheexperimentaldata.Finally,
this part ends with an introduction to statistical analysis, with a special focus on
hypothesistestingandguidelinesforselectinganappropriateomnibusstatisticaltest
fortheperformanceassessmentofmeta-heuristicstochasticoptimizationalgorithms.
Thepurposeofthesecondpartistointroducetheweaknessesofclassicstatistical
analyses that are used to compare experimental data obtained from meta-heuristic
stochasticoptimizationalgorithmsasthemotivationfordevelopingtheDeepStatis-
ticalComparison,anapproachthatprovidesmorerobuststatisticaloutcomeswhen
thereareoutlierspresentinthedataordatavaluesareinanε-neighborhood.Thispart
begins by introducing the Deep Statistical Comparison ranking scheme in a more
general form. Next, its application and extensions in single- and multi-objective
optimizationarepresentedusingexamples.
ThethirdpartfocusesontheimplementationandpracticalusageofDeepStatis-
tical Comparison approaches. It introduces the DSCTool, which is a web-service-
basede-Learningtoolforstatisticalcomparisonsofmeta-heuristicstochasticopti-
mization algorithms. The tool helps users with all Deep Statistical Comparison
approachesbyguidingthemonhowtoinputthedata(i.e.,optimizationalgorithm
results) should be organized and which statistical test is the most appropriate for
theirbenchmarkingscenario.Thispartalsoprovidesapracticalimplementationof
alltheexamplespresentedinthesecondparttoguaranteereproducibleresultsand
teachusershowtheycanusetheDeepStatisticalComparisonapproachesontheir
owndata.
Acknowledgments
Wemustthankallthefriendsandcolleagueswhosecommentsandcriticismhelped
toimprovethebook.WewouldespeciallyliketomentionBarbaraKoroušic´Seljak,
GordanaIspirova,andGjorgjinaCenikj.
WewouldalsoliketothanktheSlovenianResearchAgency(researchcorefunding
No.P2-0098,andthepostdoctoralprojectMr-BECNo.Z2-1867)andtheEuropean
Union’sHorizon2020researchandinnovationprogramundergrantagreementNo.
692286(SYNERGYforsmartmulti-objectiveoptimization)forsupportingourwork.
At the end, this is no real ending. It is just the place where one story ends and
anotheronebegins.
Ljubljana,Slovenia TomeEftimov
December2021 PeterKorošec