Automatic Algorithm Selection for Complex Simulation Problems

387 Pages·2012·3.833 MB·English

by  Roland Ewald (auth.)

Roland Ewald
Automatic Algorithm Selection for Complex Simulation Problems
VIEWEG+TEUBNER RESEARCH
Roland Ewald
Automatic Algorithm
Selection for Complex
Simulation Problems
With a foreword by Prof. Dr. Adelinde M. Uhrmacher
VIEWEG+TEUBNER RESEARCH
Bibliographic information published by the Deutsche Nationalbibliothek
The Deutsche Nationalbibliothek lists this publication in the Deutsche Nationalbibliografie;
detailed bibliographic data are available in the Internet at http://dnb.d-nb.de.
Dissertation Universität Rostock, 2010
1st Edition 2012
All rights reserved
© Vieweg+Teubner Verlag | Springer Fachmedien Wiesbaden GmbH 2012
Editorial Office: Ute Wrasmann | Anita Wilke
Vieweg+Teubner Verlag is a brand of Springer Fachmedien.
Springer Fachmedien is part of Springer Science+Business Media.
www.viewegteubner.de
No part of this publication may be reproduced, stored in a retrieval system
or transmitted, in any form or by any means, electronic, mechanical, pho-
to copying, recording, or otherwise, without the prior written permission of
the copyright holder.
Registered and/orindustrial names,trade names,trade descriptions etc.cited in this publica-
tionare part of the law for trade-mark protection and may not be used free in any form or by
any means even if this is not specifically marked.
Cover design: KünkelLopka Medienentwicklung, Heidelberg
Printed on acid-free paper
Printed in Germany
ISBN 978-3-8348-1542-2
Foreword
Simulation, anexperimentperformedwithamodel, belongstothedailyworkof
mostscientistsandpractitionersinindustryalike. Aimedatsupportingtheunder-
standing,theanalysis,and/orthedesignofcomplexdynamicsystems,simulation
belongs to the methodological toolbox of natural sciences, engineering, but also
medicine,sociology,economy,anddemography.Thediversityofapplicationareas
andintentionsofsimulationstudiesisreflectedinaplethoraofavailablemethods.
“Nosilverbulletdoesexist”—thisobservationofBrooks,referringtosoftware
engineering in general, fits also well to simulation methods. Different models,
infrastructures, and user preferences ask for different kinds of simulators. The
performance of one method, e.g., in terms of execution speed, storage consump-
tion,oraccuracyoftheresults,mightvarysignificantlyfromonesituationtothe
next. Thus,usersinterestedinperformingsimulationstudieswiththeirmodelare
faced with the problem of how to select among existing methods the most suit-
ableone,anddevelopersofsimulationmethodsarefacedwiththeproblemofhow
to evaluate the performance of their newly developed method in comparison to
others. Thosearedauntingtasks,asmostsimulationmethodsarealsohighlycon-
figurable. However,solvingthesetasksisalsoimportant,asitwilldeterminethe
qualityofsimulationstudiesandtheirresultstoalargedegree.
RolandEwald’sbookonsimulationalgorithmselectioncontributestothisquest.
It shows how methods from machine learning, portfolio theory, experiment de-
sign, adaptive software, and simulation algorithms can be combined to develop
new approaches for simulation algorithm selection. One approach exploits prior
knowledge in terms of a performance database, in which problem characteristics
and performance characteristics are stored, so that performance patterns can be
inductively learned and applied. The other approach does not depend on prior
knowledge, but learns online by reinforcement, thereby exploiting the fact that
multiplereplicationsarerequiredforstochasticsimulation. Tobeeffective, both
depend on gathering performance data, and on efficiently restricting the search
spaceofmappingsolutionstoproblems. Twocasestudies,intheareaofparallel
distributed simulation and computational systems biology, respectively, demon-
stratethepotentialofthedevelopedsolutions.
Thebookrevealsinsightsintoseveralareasofresearchandshowshowresults
canbefruitfullycombinedacrossdisciplinaryboundaries.Theconceptsdeveloped
vi Foreword
arerealizedandputtotestinaplug-inbasedmodelingandsimulationframework,
to tackle problems in concrete simulation studies. Thus, the book nicely leads
from theory to practice, and illuminates possible pitfalls along the way. It is of
relevancetoallwhoareconcernedwiththequalityofsimulationstudies,andwho
are interested in executing them in a more efficient and effective manner. The
book is also relevant for the community of researchers who develop simulation
algorithms, as it provides support for more systematic performance evaluations
andthusmorevalidperformanceresults.
Rostock,June2011 Prof. Dr. AdelindeM.Uhrmacher
Preface
IsubmittedthisthesistotheFacultyofComputerScienceandElectricalEngineer-
ing of the University of Rostock in August 2010, after working on this topic for
morethanfouryears.1 Manypeoplesupportedmeduringthistime.
IwanttostartwiththankingmysupervisorLinUhrmacherforherenduringen-
couragementandguidance. Thesamegoesforallcurrentandformerfriendsand
colleaguesfromthemodelingandsimulationgroupattheUniversityofRostock;
it was great fun to work with you and I learned a lot! You know how dubious I
findmostrankings—andhowtovalueyourimmeasurablesupport?—sohereyou
areinalphabeticalorder: AlexanderSteiniger,AlkeMartens,AnjaHampel,Arne
Bittig,CarstenMaus,FieteHaack,FlorianMarquardt,JanHimmelspach,Mathias
John, Mathias Röhl, Matthias Jeschke, Nadja Schlungbaum, Orianne Mazemon-
det, Sigrun Hoffmann, Stefan Leye, Stefan Rybacki, and Susanne Jürgensmann.
Andlet’snotforgetFritz;-)
Special thanks go to Stefan Leye, who took over teaching one of my exercise
groupsduringthelastsemesterofwritingthis,toMatthiasJeschke,whoprovided
many interesting SSA algorithms for evaluation, and to my officemate Jan Him-
melspach. IwouldalsoliketothankBingWang,whodevelopedthePDESalgo-
rithmsIevaluatedinchapter10(p.303).KaustavSahaandSteffenTorbahnhelped
mewithimplementingsomepartsoftheSPDM;RenéSchulzimplementedsome
of the multi-armed bandit policies and worked with me on the genetic algorithm
forportfolioselection(sec.7.2,p.208).
Whilesucceedingatworkisonething,stayingsanewhilegoingthroughthisis
another. I am deeply grateful to my family; for their support, their patience, and
theirunderstanding.
Rostock,June2011 RolandEwald
1Prototypicalimplementationsofthedevelopedmethodshavebeenrealizedfortheopen-sourcemod-
elingandsimulationframeworkJAMESII((cid:0)(cid:2)(cid:2)(cid:3)(cid:4)(cid:5)(cid:5)(cid:6)(cid:7)(cid:8)(cid:9)(cid:10)(cid:11)(cid:11)(cid:12)(cid:13)(cid:14)(cid:15)).
Contents
Foreword v
ListofFigures xv
ListofTables xix
ListofListings xxi
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Terminology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3.1 SimulationofChemicalReactionNetworks . . . . . . . . 7
1.3.2 ParallelandDistributedDiscrete-EventSimulation . . . . 9
1.4 EpistemologicalViewpoint . . . . . . . . . . . . . . . . . . . . . 13
1.5 Structure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
I Background 17
2 AlgorithmSelection 19
2.1 TheAlgorithmSelectionProblem . . . . . . . . . . . . . . . . . 19
2.1.1 ImportantSub-Problems . . . . . . . . . . . . . . . . . . 21
2.1.2 EffectivenessandEfficiency . . . . . . . . . . . . . . . . 24
2.1.3 FurtherASPProperties . . . . . . . . . . . . . . . . . . . 29
2.1.4 ASPinaSimulationContext . . . . . . . . . . . . . . . . 32
2.2 AnalyticalAlgorithmSelection . . . . . . . . . . . . . . . . . . . 33
2.3 AlgorithmSelectionasLearning . . . . . . . . . . . . . . . . . . 36
2.3.1 ErrorSources,ErrorTypes,Bias-VarianceTrade-Off . . . 38
2.3.2 ReinforcementLearning . . . . . . . . . . . . . . . . . . 43
2.3.3 FurtherAspectsofLearning . . . . . . . . . . . . . . . . 50
2.4 AlgorithmSelectionasAdaptationtoComplexity . . . . . . . . . 52
2.4.1 ComplexSimulationProblems . . . . . . . . . . . . . . . 52
x Contents
2.4.2 ComplexAdaptiveSystems . . . . . . . . . . . . . . . . 53
2.4.3 Self-AdaptiveSoftwareandAutonomousComputing . . . 55
2.5 AlgorithmPortfolios . . . . . . . . . . . . . . . . . . . . . . . . 58
2.5.1 IdentifyingEfficientPortfolios . . . . . . . . . . . . . . . 60
2.5.2 FromFinancialtoAlgorithmicPortfolios . . . . . . . . . 61
2.5.3 AlgorithmPortfolioVariants . . . . . . . . . . . . . . . . 64
2.5.4 PortfoliosforSimulationAlgorithmSelection . . . . . . . 68
2.6 CategorizationofAlgorithmSelectionMethods . . . . . . . . . . 71
2.6.1 CategorizationAspects . . . . . . . . . . . . . . . . . . . 72
2.6.2 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . 78
2.7 ApplicationsofAlgorithmSelection . . . . . . . . . . . . . . . . 80
2.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
3 SimulationAlgorithmPerformanceAnalysis 93
3.1 ChallengesinExperimentalAlgorithmics . . . . . . . . . . . . . 93
3.1.1 EfficientImplementationsandComparability . . . . . . . 94
3.1.2 Reproducibility . . . . . . . . . . . . . . . . . . . . . . . 96
3.1.3 SimulationExperimentDescriptions . . . . . . . . . . . . 100
3.2 ExperimentDesign . . . . . . . . . . . . . . . . . . . . . . . . . 101
3.2.1 VarianceReduction . . . . . . . . . . . . . . . . . . . . . 101
3.2.2 Optimization,SensitivityAnalysis,andMeta-Modeling . 104
3.2.3 FurtherAspectsofPerformanceExperiments . . . . . . . 106
3.3 SimulatorPerformanceAnalysisandPrediction . . . . . . . . . . 108
3.3.1 AnalyticalMethods . . . . . . . . . . . . . . . . . . . . . 108
3.3.2 EmpiricalMethods . . . . . . . . . . . . . . . . . . . . . 112
3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
II MethodsandImplementation 117
4 AFrameworkforSimulationAlgorithmSelection 119
4.1 RequirementsAnalysis: UseCases . . . . . . . . . . . . . . . . . 119
4.2 BriefIntroductiontoJAMESII . . . . . . . . . . . . . . . . . . 122
4.2.1 Fundamentals . . . . . . . . . . . . . . . . . . . . . . . . 122
4.2.2 RelationtoSelf-AdaptiveSoftware . . . . . . . . . . . . 131
4.2.3 LimitationsofAlgorithmSelectioninJAMESII . . . . . 132
4.3 TechnicalRequirementsforAlgorithmSelectioninJAMESII . . 134
4.4 ASimulationAlgorithmSelectionFramework . . . . . . . . . . . 139
4.4.1 RelatedSoftwareSystems . . . . . . . . . . . . . . . . . 140