Clever Algorithms

454 Pages·2013·1.81 MB·English

Jason Brownlee
Clever Algorithms
Nature-Inspired Programming Recipes
ii
Jason Brownlee, PhD
Jason Brownlee studied Applied Science at Swinburne University in
Melbourne, Australia, going on to complete a Masters in Information
Technology focusing on Niching Genetic Algorithms, and a PhD in the field
of Artificial Immune Systems. Jason has worked for a number of years as a
Consultant and Software Engineer for a range of Corporate and Government
organizations. When not writing books, Jason likes to compete in Machine
Learning competitions.
Cover Image
© Copyright 2011 Jason Brownlee. All Reserved.
Clever Algorithms: Nature-Inspired Programming Recipes
© Copyright 2011 Jason Brownlee. Some Rights Reserved.
Revision 2. 16 June 2012
ISBN: 978-1-4467-8506-5
This work is licensed under a Creative Commons
Attribution-Noncommercial-Share Alike 2.5 Australia License.
The full terms of the license are located online at
http://creativecommons.org/licenses/by-nc-sa/2.5/au/legalcode
Webpage
Source code and additional resources can be downloaded from the books
companion website online at http://www.CleverAlgorithms.com
Contents
Foreword vii
Preface ix
I Background 1
1 Introduction 3
1.1 What is AI . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Problem Domains. . . . . . . . . . . . . . . . . . . . . . 10
1.3 Unconventional Optimization . . . . . . . . . . . . . . . 13
1.4 Book Organization . . . . . . . . . . . . . . . . . . . . . 17
1.5 How to Read this Book . . . . . . . . . . . . . . . . . . 20
1.6 Further Reading . . . . . . . . . . . . . . . . . . . . . . 21
1.7 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . 22
II Algorithms 27
2 Stochastic Algorithms 29
2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.2 Random Search . . . . . . . . . . . . . . . . . . . . . . . 30
2.3 Adaptive Random Search . . . . . . . . . . . . . . . . . 34
2.4 Stochastic Hill Climbing . . . . . . . . . . . . . . . . . . 40
2.5 Iterated Local Search. . . . . . . . . . . . . . . . . . . . 44
2.6 Guided Local Search . . . . . . . . . . . . . . . . . . . . 50
2.7 Variable Neighborhood Search. . . . . . . . . . . . . . . 57
2.8 Greedy Randomized Adaptive Search. . . . . . . . . . . 63
2.9 Scatter Search . . . . . . . . . . . . . . . . . . . . . . . 69
2.10 Tabu Search . . . . . . . . . . . . . . . . . . . . . . . . . 76
2.11 Reactive Tabu Search . . . . . . . . . . . . . . . . . . . 82
iii
iv Contents
3 Evolutionary Algorithms 91
3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 91
3.2 Genetic Algorithm . . . . . . . . . . . . . . . . . . . . . 96
3.3 Genetic Programming . . . . . . . . . . . . . . . . . . . 103
3.4 Evolution Strategies . . . . . . . . . . . . . . . . . . . . 113
3.5 Differential Evolution . . . . . . . . . . . . . . . . . . . 119
3.6 Evolutionary Programming . . . . . . . . . . . . . . . . 125
3.7 Grammatical Evolution . . . . . . . . . . . . . . . . . . 131
3.8 Gene Expression Programming . . . . . . . . . . . . . . 139
3.9 Learning Classifier System . . . . . . . . . . . . . . . . . 146
3.10 Non-dominated Sorting Genetic Algorithm. . . . . . . . 157
3.11 Strength Pareto Evolutionary Algorithm . . . . . . . . . 165
4 Physical Algorithms 173
4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 173
4.2 Simulated Annealing . . . . . . . . . . . . . . . . . . . . 175
4.3 Extremal Optimization. . . . . . . . . . . . . . . . . . . 181
4.4 Harmony Search . . . . . . . . . . . . . . . . . . . . . . 188
4.5 Cultural Algorithm . . . . . . . . . . . . . . . . . . . . . 194
4.6 Memetic Algorithm . . . . . . . . . . . . . . . . . . . . . 200
5 Probabilistic Algorithms 207
5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 207
5.2 Population-Based Incremental Learning . . . . . . . . . 211
5.3 Univariate Marginal Distribution Algorithm . . . . . . . 216
5.4 Compact Genetic Algorithm . . . . . . . . . . . . . . . . 220
5.5 Bayesian Optimization Algorithm . . . . . . . . . . . . . 224
5.6 Cross-Entropy Method . . . . . . . . . . . . . . . . . . . 232
6 Swarm Algorithms 237
6.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 237
6.2 Particle Swarm Optimization . . . . . . . . . . . . . . . 241
6.3 Ant System . . . . . . . . . . . . . . . . . . . . . . . . . 247
6.4 Ant Colony System . . . . . . . . . . . . . . . . . . . . . 254
6.5 Bees Algorithm . . . . . . . . . . . . . . . . . . . . . . . 261
6.6 Bacterial Foraging Optimization Algorithm . . . . . . . 266
7 Immune Algorithms 275
7.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 275
7.2 Clonal Selection Algorithm . . . . . . . . . . . . . . . . 280
7.3 Negative Selection Algorithm . . . . . . . . . . . . . . . 288
7.4 Artificial Immune Recognition System . . . . . . . . . . 295
7.5 Immune Network Algorithm . . . . . . . . . . . . . . . . 304
7.6 Dendritic Cell Algorithm. . . . . . . . . . . . . . . . . . 312
v
8 Neural Algorithms 319
8.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 319
8.2 Perceptron . . . . . . . . . . . . . . . . . . . . . . . . . 323
8.3 Back-propagation . . . . . . . . . . . . . . . . . . . . . . 328
8.4 Hopfield Network . . . . . . . . . . . . . . . . . . . . . . 336
8.5 Learning Vector Quantization . . . . . . . . . . . . . . . 342
8.6 Self-Organizing Map . . . . . . . . . . . . . . . . . . . . 348
III Extensions 357
9 Advanced Topics 359
9.1 Programming Paradigms . . . . . . . . . . . . . . . . . . 360
9.2 Devising New Algorithms . . . . . . . . . . . . . . . . . 370
9.3 Testing Algorithms . . . . . . . . . . . . . . . . . . . . . 382
9.4 Visualizing Algorithms . . . . . . . . . . . . . . . . . . . 389
9.5 Problem Solving Strategies . . . . . . . . . . . . . . . . 402
9.6 Benchmarking Algorithms . . . . . . . . . . . . . . . . . 417
IV Appendix 427
A Ruby: Quick-Start Guide 429
A.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 429
A.2 Language Basics . . . . . . . . . . . . . . . . . . . . . . 429
A.3 Ruby Idioms . . . . . . . . . . . . . . . . . . . . . . . . 433
A.4 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . 435
Errata 437
Index 439
vi Contents
Foreword
I am delighted to write this foreword. This book, a reference where
one can look up the details of most any algorithm to find a clear
unambiguous description, has long been needed and here it finally is.
A concise reference that has taken many hours to write but which has
the capacity to save vast amounts of time previously spent digging out
original papers.
I have known the author for several years and have had experience
of his amazing capacity for work and the sheer quality of his output, so
this book comes as no surprise to me. But I hope it will be a surprise
and delight to you, the reader for whom it has been written.
But useful as this book is, it is only a beginning. There are so many
algorithms that no one author could hope to cover them all. So if you
know of an algorithm that is not yet here, how about contributing it
using the same clear and lucid style?
Professor Tim Hendtlass
Complex Intelligent Systems Laboratory
Faculty of Information and Communication Technologies
Swinburne University of Technology
Melbourne, Australia
2010
vii
viii Foreword
Preface
About the book
The need for this project was born of frustration while working to-
wards my PhD. I was investigating optimization algorithms and was
implementing a large number of them for a software platform called
the Optimization Algorithm Toolkit (OAT)1. Each algorithm required
considerable effort to locate the relevant source material (from books,
papers, articles, and existing implementations), decipher and interpret
the technique, and finally attempt to piece together a working imple-
mentation.
Taking a broader perspective, I realized that the communication of
algorithmic techniques in the field of Artificial Intelligence was clearly a
difficult and outstanding open problem. Generally, algorithm descrip-
tions are:
• Incomplete: many techniques are ambiguously described, partially
described, or not described at all.
• Inconsistent: a given technique may be described using a variety
of formal and semi-formal methods that vary across different tech-
niques,limitingthetransferabilityofbackgroundskillsanaudience
requires to read a technique (such as mathematics, pseudocode,
program code, and narratives). An inconsistent representation for
techniques means that the skills used to understand and internal-
ize one technique may not be transferable to realizing different
techniques or even extensions of the same technique.
• Distributed: the description of data structures, operations, and
parameterization of a given technique may span a collection of
papers, articles, books, and source code published over a number
of years, the access to which may be restricted and difficult to
obtain.
1OATlocatedathttp://optalgtoolkit.sourceforge.net
ix
x Preface
For the practitioner, a badly described algorithm may be simply
frustrating, where the gaps in available information are filled with
intuition and ‘best guess’. At the other end of the spectrum, a badly
describedalgorithmmaybeanexampleofbadscienceandthefailureof
the scientific method, where the inability to understand and implement
a technique may prevent the replication of results, the application, or
the investigation and extension of a technique.
ThesoftwareIproducedprovidedafirststepsolutiontothisproblem:
asetofworkingalgorithmsimplementedina(somewhat)consistentway
and downloaded from a single location (features likely provided by any
libraryofartificialintelligencetechniques). Thenextlogicalstepneeded
to address this problem is to develop a methodology that anybody can
follow. The strategy to address the open problem of poor algorithm
communication is to present complete algorithm descriptions (rather
than just implementations) in a consistent manner, and in a centralized
location. Thisbookistheoutcomeofdevelopingsuchastrategythatnot
onlyprovidesamethodologyforstandardizedalgorithmdescriptions,but
providesalargecorpusofcompleteandconsistentalgorithmdescriptions
in a single centralized location.
The algorithms described in this work are practical, interesting, and
fun,andthegoalofthisprojectwastopromotethesefeaturesbymaking
algorithms from the field more accessible, usable, and understandable.
Thisprojectwasdevelopedoveranumberyearsthroughalotofwriting,
discussion,andrevision. Thisbookhasbeenreleasedunderapermissive
license that encourages the reader to explore new and creative ways of
further communicating its message and content.
Ihopethatthisprojecthassucceededinsomesmallwayandthatyou
too can enjoy applying, learning, and playing with Clever Algorithms.
Jason Brownlee
Melbourne, Australia
2011