Genetic Algorithms + Data Structures = Evolution Programs

257 Pages·1992·4.59 MB·English

by  Zbigniew Michalewicz (auth.)

Artificial Intelligence
Managing Editor: D. W. Loveland
Editors: S. Amarel A. Biermann L. Bole A. Bundy
H. Ga1laire P. Hayes A. Joshi D. Lenat M. A. Me Robbie
A. Maekworth D. Nau R. Reiter E. Sandewa11 S. Shafer
Y. Shoham J. Siekmann W. Wahlster
Zbigniew Michalewicz
Genetic Algorithms
+l)ataStructures
=
Evolution Programs
With 48 Figures
Springer-Verlag
Berlin Heidelberg GmbH
Zbigniew Michalewicz
Department of Computer Science
University of North Carolina
Charlotte, NC 28223, USA
ISBN 978-3-662-02832-2
Library of Congress Cataloging-in-Publication Data
Michalewicz, Zbigniew. Genetic algorithms + data structures = evolution programs 1
Zbigniew Michalewicz. p. cm.-(Artificial intelligence) Includes bibliographical refer
ences and index.
ISBN 978-3-662-02832-2 ISBN 978-3-662-02830-8 (eBook)
DOI 10.1007/978-3-662-02830-8
Computer algorithms. 2. Data structures
(Computer science) 3. Computer programs. 1. Title II. Series.
QA76.9.A43M53 1992 005.1-dc20 92-19925
This work is subject to copyright. Ali rights are reserved, whether the whole or part of the
material is concemed, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data
banks. Duplication of this publication or parts thereof is permitted only under the provisions
of the German Copyright Law of September 9, 1965, in its current version, and permission
for use must always be obtained from Springer-Verlag Berlin Heidelberg GmbH .
Violations are liable for prosecution under the German Copyright Law.
© Springer-Verlag Berlin Heidelberg 1992
Originally pub1ished by Springer-Ve r1ag Berlin Heide1berg New York in 1992
Softcover reprint of the hardcover 1s t edition 1992
The use of general descriptive names, registered names, trademarks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from
the relevant protective laws and regulations and therefore free for general use.
Typesetting: Camera ready by author
45/3140-5 4 3 2 1 O-Printed on acid-free paper
To the next generation:
Matthew, Katherine, Michael,
Thomas, and Irene
Preface
'What does your Master teach?'
asked a visitor.
'Nothing,' said the disciple.
'Then why does he give discourses?'
'He only points the way - he teaches
nothing.'
Anthony de Mello, One Minute Wisdom
During the last three decades there has been a growing interest in algorithms
which rely on analogies to natural processes. The emergence of massively par
allel computers made these algorithms of practical interest. The best known
algorithms in this class include evolutionary programming, genetic algorithms,
evolution strategies, simulated annealing, classifier systems, and neural net
works. Recently (1-3 October 1990) the University of Dortmund, Germany,
hosted the First Workshop on Parallel Problem Solving from Nature [164].
This book discusses a subclass of these algorithms - those which are based
on the principle of evolution (survival of the fittest). In such algorithms a popu
lation of individuals (potential solutions) undergoes a sequence of unary (muta
tion type) and higher order (crossover type) transformations. These individuals
strive for survival: a selection scheme, biased towards fitter individuals, selects
the next generation. After some number of generations, the program converges
- the best individual hopefully represents the optimum solution.
There are many different algorithms in this category. To underline the sim
ilarities between them we use the common term "evolution programs" .
Evolution programs can be perceived as a generalization of genetic al
gorithms. Classical genetic algorithms operate on fixed-length binary strings,
which need not be the case for evolution programs. Also, evolution programs
usually incorporate a variety of "genetic" operators, whereas classical genetic
algorithms use binary crossover and mutation.
The beginnings of genetic algorithms can be traced back to the early 1950s
when several biologists used computers for simulations of biological systems
[72]. However, the work done in late 1960s and early 1970s at the University
of Michigan under the direction of John Holland led to genetic algorithms as
they are known today. The interest in genetic algorithms is growing rapidly -
the recent Fourth International Conference on Genetic Algorithms (San Diego,
13-16 July 1991) attracted around 300 participants.
The book describes the results of three years' research from early 1989 at
Victoria University of Wellington, New Zealand, through late 1991 (since July
1989 I have been at the University of North Carolina at Charlotte). During this
time I built and experimented with various modifications of genetic algorithms
VIII Preface
using different data structures for chromosome representation of individuals,
and various "genetic" operators which operated on them. Because of my back
ground in databases [122], [126], where the constraints playa central role, most
evolution programs were developed for constrained problems.
The idea of evolution programs (in the sense presented in this book), was
conceived quite early [101], [133] and was supported later by a series of ex
periments. Despite the fact that evolution programs, in general, lack a strong
theoretical background, the experimental results were more than encouraging:
very often they performed much better than classical genetic algorithms, than
commercial systems, and than other, best-known algorithms for a particular
class of problems.
Some other researchers, at different stages of their research, performed ex
periments which were perfect examples of the "evolution programming" tech
nique - some of them are discussed in this volume. Chapter 8 presents a survey
of evolution strategies - a technique developed in Germany by I. Rechenberg
and H.-P. Schwefel [149], [162] for parameter optimization problems. Many re
searchers investigated the properties of evolution systems for ordering problems,
including the widely known, "traveling salesman problem" (Chapter 10). In
Chapter 11 we present systems for a variety of problems including problems on
graphs, scheduling, and partitioning. Chapter 12 describes the construction of
an evolution program for inductive learning in attribute based spaces, developed
by C. Janikow [97]. In the Conclusions, we briefly discuss evolution programs
for generating LISP code to solve problems, developed by J. Koza [108], and
present an idea for a new programming environment.
The book is organized as follows. The introduction provides a general discus
sion on the motivation and presents the main idea of the book. Since evolution
programs are based on the principles of genetic algorithms, Part I of this book
serves as survey on this topic. We explain what genetic algorithms are, how they
work, and why (Chapters 1-3). The last chapter of Part I (Chapter 4) presents
some selected issues (selection routines, scaling, etc.) for genetic algorithms.
In Part II we explore a single data structure: a vector in a floating point
representation, only recently widely accepted in the GA community [38]. We
talk only about numerical optimization. We present some experimental com
parison of binary and floating point representations (Chapter 5) and discuss
new "genetic" operators responsible for fine local tuning (Chapter 6). Chapter
7 presents two evolution programs to handle constrained problems: the GENO
COP system, for optimizing functions in the presence of linear constraints, and
the GAFOC system, for optimal control problems. Various tests cases are con
sidered; the results of the evolution programs are compared with a commercial
system. The last chapter of this part (Chapter 8) presents a survey of evolution
strategies and describes some other methods.
Part III discusses a collection of evolution programs built over recent years
to explore their applicability to a variety of hard problems. We present further
experiments with order-based evolution programs, evolution programs with ma-
Preface IX
trices and graphs as a chromosome structure. Also, we discuss an application
of evolution program to machine learning, comparing it with other approaches.
The title of this book rephrases the famous expression used by N. Wirth
fifteen years ago for his book, Algorithms + Data Structures = Programs [197].
Both books share a common idea. To build a successful program (in particular,
an evolution program), appropriate data structures should be used (the data
structures, in the case of the evolution program, correspond to the chromosome
representation) together with appropriate algorithms (these correspond to "ge
netic" operators used for transforming one or more individual chromosomes).
The book is aimed at a large audience: graduate students, programmers, re
searchers, engineers, designers - everyone who faces challenging optimization
problems. In particular, the book will be of interest to the Operations Research
community, since many of the problems considered (traveling salesman, schedul
ing, transportation problems) are from their area of interest. An understanding
of introductory college level mathematics and of basic concepts of programming
is sufficient to follow all presented material.
Acknowledgements
It is a pleasure to acknowledge the assistance of several people and institutions
in this effort.
Thanks are due to many co-authors who worked with me at different stages
of this project; these are (in alphabetical order): Paul Elia, Lindsay Groves,
Matthew Hobbs, Cezary Janikow, Andrzej Jankowski, Mohammed Kazemi,
Jacek Krawczyk, Zbigniew Ras, Joseph Schell, David Seniv, Don Shoff, Jim
Stevens, Tony Vignaux, and George Windholz.
I would like to thank all my graduate students who took part in a course
Genetic Algorithms I offered at UNC-Charlotte in Fall 1990, and my Mas
ter students, who wrote their thesis on various modifications of genetic algo
rithms: Jason Foodman, Jay Livingstone, Jeffrey B. Rayfield, David Seniw, Jim
Stevens, Charles Strickland, Swarnalatha Swaminathan, 'l-nd Keith Wharton.
Jim Stevens provided a short story on rabbits and foxes (Chapter 1).
Thanks are due to Abdollah Homaifar from North Carolina State University,
Kenneth Messa from Loyola University (New Orleans), and to several colleagues
at UNC-Charlotte: Mike Allen, Rick Lejk, Zbigniew Ras, Harold Reiter, Joe
Schell, and Barry Wilkinson, for reviewing selected parts of the book.
I would like to acknowledge the assistance of Doug Gullett, Jerry Holt,
and Dwayne McNeil (Computing Services, UNC-Charlotte) for recovering all
chapters of the book (accidentally deleted). I would like also to thank Jan
Thomas (Academic Computing Services, UNC-Charlotte) for overall help.
I would like to extend my thanks to the editors of the Artificial Intelligence
Series, Springer-Verlag: Leonard Bolc, Donald Loveland, and Hans Wossner for
initiating the idea of the book and their help throughout the project. Special
thanks are due to J. Andrew Ross, the English Copy Editor, Springer-Verlag,
for his precious assistance with writing style.
X Preface
I would like to acknowledge a series of grants from North Carolina Su
percomputing Center (1990-1991), which allowed me to run the hundreds of
experiments described in this text.
In the book I have included many citations from other published work, as
well as parts of my own published articles. Therefore, I would like to acknowledge
all permissions to use such material in this publication I received from the
following publishers: Pitman Publishing Company; Birkhiiuser Verlag AG; John
Wiley and Sons, Ltd; Complex Systems Publications, Inc.; Addison-Wesley
Publishing Company; Kluwer Academic Publishers; Chapman and Hall Ltd,
Scientific, Technical and Medical Publishers; Prentice Hall; IEEE; Association
for Computing Machinery; Morgan Kaufmann Publishers, Inc.; and individuals:
David Goldberg, Fred Glover, John Grefenstette, Abdollah Homaifar, and John
Koza.
Finally, I would like to thank my family for their patience and support
during the (long) summer of 1991.
Charlotte Zbigniew Michalewicz
January 1992
Table of Contents
Introduction
Part I. Genetic Algorithms 11
1 GAs: What Are They? . . . . . 13
1.1 Optimization of a simple function 18
1.1.1 Representation .. . 19
1.1.2 Initial population . . 20
1.1.3 Evaluation function. 20
1.1.4 Genetic operators . . 21
1.1.5 Parameters ..... 21
1.1.6 Experimental results 22
1.2 The prisoner's dilemma. . . 22
1.2.1 Representing a strategy 23
1.2.2 Outline of the genetic algorithm . 23
1.2.3 Experimental results . . . . . . . 24
1.3 Traveling salesman problem . . . . . . . 25
1.4 Hill climbing, simulated annealing, and genetic algorithms 26
1.5 Conclusions . . . . . . 30
2 GAs: How Do They Work? 31
3 GAs: Why Do They Work? 43
4 GAs: Selected Topics 55
4.1 Sampling mechanism . 56
4.2 Characteristics of the function 62
4.3 Other ideas . . . . . . . . . . 67
Part II. Numerical Optimization 73
5 Binary or Float? ....... . 75
5.1 The test case ...... . 76
5.2 The two implementations. 77
5.2.1 The binary implementation 77
5.2.2 The floating point implementation 77
5.3 The experiments ............ . 78
5.3.1 Random mutation and crossover. 78
5.3.1.1 Binary ......... . 78