Iterative Computer Algorithms with Applications in Engineering: Solving Combinatorial Optimization Problems

398 Pages·2000·2.079 MB·English

by  Sadiq M. Sait, Habib Youssef

Iterative Computer Algorithms with Applications
in Engineering
(Solving Combinatorial Optimization Problems )
Sadiq M. Sait and Habib Youssef
2
Sadiq M. Sait (S’87, M’90) obtained a Bachelor’s degree in Electronics
from Bangalore University, India, in 1981, and Master’s and Ph.D degrees in
Electrical Engineering from King Fahd University of Petroleum and Minerals
(KFUPM),Dhahran,in1983and1987respectively. HeiscurrentlyaProfessor,
intheDepartmentofComputerEngineeringofKFUPM.Saithasauthoredover
85 research papers in International Journals and Conferences. He is the co-
author of the book ”VLSI Physical Design Automation: Theory and Practice”,
published by McGraw-Hill Book Co., Europe, (also co-published by IEEE
Press), January 1995. He has also contributed two chapters to a book entitled
‘Progress in VLSI Design’. He served on the editorial board of International
Journal of Computer-Aided Design between 1988-1990. Currently he is the
editor of ‘Arabian Journal for Science and Engineering’ for Computer Science
& Engineering. His current areas of interest are in Digital Design Automation,
VLSI System Design, High-Level Synthesis, and Iterative Algorithms.
Email: [email protected].
Habib Youssef (S’86, M’90) received a Diplome d’Ingenieur en Informatique
fromtheFacultedesSciencesdeTunis,Tunisiain1982andaPh.D.incomputer
science from the University of Minnesota in 1990. He is currently an Associate
Professor of Computer Engineering at King Fahd University of Petroleum and
Minerals, Saudi Arabia. Habib Youssef has authored more than 45 journal and
conference papers. He is the co-author of the book ”VLSI Physical Design
Automation: Theory and Practice”, published by McGraw-Hill Book Co.,
Europe, (also co-published by IEEE Press), January 1995. His main research
interests are CAD of VLSI, Computer networks, and performance evaluation of
computer systems, and general stochastic and evolutionary algorithms.
Email: [email protected].
3
Dedication
To
Our wives Sumaiya and Leila
Whose Care, Patience, & Understanding Made It Possible.
Sadiq M. Sait & Habib Youssef
This book represents a joint effort of both authors who closely cooperated on
allchapters. Thenamesofauthorshavebeenlistedinalphabeticalorders. This
does not imply any senior-junior relationship.
4
Contents
1 Introduction 3
1.1 Combinatorial Optimization . . . . . . . . . . . . . . . . . . . . . 3
1.1.1 Complexity of Algorithms . . . . . . . . . . . . . . . . . . 7
1.1.2 Hard Problems versus Easy Problems . . . . . . . . . . . 8
1.2 Optimization Methods . . . . . . . . . . . . . . . . . . . . . . . . 11
1.3 States, Moves, and Optimality . . . . . . . . . . . . . . . . . . . 13
1.4 Local Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.4.1 Deterministic and Stochastic Algorithms . . . . . . . . . . 16
1.5 Optimal versus Final Solution . . . . . . . . . . . . . . . . . . . . 17
1.6 Single versus Multi-criteria Constrained Optimization . . . . . . 18
1.7 Convergence Analysis of Iterative Algorithms . . . . . . . . . . . 20
1.7.1 Configuration Graph . . . . . . . . . . . . . . . . . . . . . 20
1.8 Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
1.8.1 Time-Homogeneous Markov Chains . . . . . . . . . . . . 22
1.8.2 Perturbation Probability. . . . . . . . . . . . . . . . . . . 23
1.8.3 Ergodic Markov Chains . . . . . . . . . . . . . . . . . . . 23
1.8.4 Acceptance Probability . . . . . . . . . . . . . . . . . . . 27
1.8.5 Transition Probability . . . . . . . . . . . . . . . . . . . . 27
1.9 Parallel Processing . . . . . . . . . . . . . . . . . . . . . . . . . . 28
1.10 Summary and Organization of the Book . . . . . . . . . . . . . . 34
2 Simulated Annealing (SA) 53
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
2.1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.2 Simulated Annealing Algorithm . . . . . . . . . . . . . . . . . . . 57
2.3 SA Convergence Aspects . . . . . . . . . . . . . . . . . . . . . . . 60
2.4 Parameters of the SA Algorithm . . . . . . . . . . . . . . . . . . 70
2.4.1 A Simple Cooling Schedule . . . . . . . . . . . . . . . . . 71
2.4.2 A Statistical Polynomial Time Cooling Schedule . . . . . 72
2.4.3 An Efficient General Cooling Schedule . . . . . . . . . . . 75
2.5 SA Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
2.6 SA Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
2.6.1 VLSI placement & TimberWolf3.2 . . . . . . . . . . . . . 79
2.6.2 A Brief Overview of Other Applications . . . . . . . . . . 84
5
6 CONTENTS
2.7 Parallelization of SA . . . . . . . . . . . . . . . . . . . . . . . . . 84
2.7.1 Parallel Annealing . . . . . . . . . . . . . . . . . . . . . . 85
2.7.2 Multiple-trials Parallelism . . . . . . . . . . . . . . . . . . 88
2.8 Conclusions and Recent Work . . . . . . . . . . . . . . . . . . . . 94
3 Genetic Algorithms (GAs) 111
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
3.1.1 GA Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
3.1.2 GA Terminology . . . . . . . . . . . . . . . . . . . . . . . 113
3.2 Genetic Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 120
3.3 Schema Theorem and Implicit Parallelism . . . . . . . . . . . . . 122
3.3.1 Schema Theorem . . . . . . . . . . . . . . . . . . . . . . . 123
3.3.2 Implicit Parallelism . . . . . . . . . . . . . . . . . . . . . 126
3.4 GA Convergence Aspects . . . . . . . . . . . . . . . . . . . . . . 127
3.4.1 GA without Mutation or Inversion . . . . . . . . . . . . . 128
3.4.2 GA with Crossover and Mutation . . . . . . . . . . . . . . 129
3.5 GA In Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
3.5.1 Genetic Operators . . . . . . . . . . . . . . . . . . . . . . 133
3.5.2 Scoring/Fitness function . . . . . . . . . . . . . . . . . . . 139
3.5.3 Scaling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
3.5.4 Selection of Parents . . . . . . . . . . . . . . . . . . . . . 141
3.6 GA Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
3.6.1 Population . . . . . . . . . . . . . . . . . . . . . . . . . . 143
3.6.2 Generation Gap . . . . . . . . . . . . . . . . . . . . . . . 144
3.6.3 Operator Probabilities . . . . . . . . . . . . . . . . . . . . 145
3.6.4 Other Issues. . . . . . . . . . . . . . . . . . . . . . . . . . 146
3.7 GA Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
3.7.1 Traveling Salesman Problem . . . . . . . . . . . . . . . . 148
3.7.2 Module Placement . . . . . . . . . . . . . . . . . . . . . . 152
3.7.3 Applications of GAs to Other Problems . . . . . . . . . . 155
3.8 Parallelization of GA . . . . . . . . . . . . . . . . . . . . . . . . 157
3.8.1 Island model . . . . . . . . . . . . . . . . . . . . . . . . . 158
3.8.2 Stepping stone model . . . . . . . . . . . . . . . . . . . . 158
3.8.3 Neighborhood or cellular model . . . . . . . . . . . . . . . 158
3.9 Other Issues and Recent Work . . . . . . . . . . . . . . . . . . . 160
3.9.1 Performance Measurement. . . . . . . . . . . . . . . . . . 160
3.9.2 Gray vs Binary Encoding . . . . . . . . . . . . . . . . . . 160
3.9.3 Niches, Crowding and Speciation . . . . . . . . . . . . . . 161
3.9.4 Constant vs Dynamically Decreasing Population . . . . . 163
3.9.5 Multi-Objective Optimization with GAs . . . . . . . . . . 164
3.9.6 Other Related Material . . . . . . . . . . . . . . . . . . . 164
3.10 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
CONTENTS 7
4 Tabu Search (TS) 185
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
4.2 Tabu Search Algorithm . . . . . . . . . . . . . . . . . . . . . . . 187
4.3 Implementation Related Issues . . . . . . . . . . . . . . . . . . . 195
4.3.1 Move Attributes . . . . . . . . . . . . . . . . . . . . . . . 195
4.3.2 Tabu List and Tabu Restrictions . . . . . . . . . . . . . . 196
4.3.3 Data Structure to Handle Tabu-lists . . . . . . . . . . . . 199
4.3.4 Other Aspiration Criteria . . . . . . . . . . . . . . . . . . 205
4.4 Limitations of Short-Term Memory . . . . . . . . . . . . . . . . . 207
4.4.1 Intermediate-Term Memory (Search Intensification) . . . 208
4.4.2 Long-Term Memory (Search Diversification) . . . . . . . . 209
4.5 Examples of Diversifying Search . . . . . . . . . . . . . . . . . . 212
4.5.1 Diversification in Quadratic Assignment (QAP) . . . . . . 212
4.5.2 Diversification in Multiprocessor Scheduling . . . . . . . 215
4.6 TS Convergence Aspects . . . . . . . . . . . . . . . . . . . . . . . 216
4.7 TS Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
4.7.1 QAP Problem . . . . . . . . . . . . . . . . . . . . . . . . 219
4.7.2 Bandwidth Packing Problem (BWP) . . . . . . . . . . . . 222
4.8 Parallelization of TS . . . . . . . . . . . . . . . . . . . . . . . . . 224
4.9 Other Issues and Related Work . . . . . . . . . . . . . . . . . . . 230
4.9.1 Neighborhood, Evaluator functions, Stopping Criteria . . 230
4.9.2 Target Analysis (TA) . . . . . . . . . . . . . . . . . . . . 232
4.9.3 Candidate List Strategies . . . . . . . . . . . . . . . . . . 236
4.9.4 Strategic Oscillation . . . . . . . . . . . . . . . . . . . . . 239
4.9.5 Path Relinking . . . . . . . . . . . . . . . . . . . . . . . . 241
4.10 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
5 Simulated Evolution (SimE) 255
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
5.2 Historical Background . . . . . . . . . . . . . . . . . . . . . . . . 256
5.3 Simulated Evolution Algorithm . . . . . . . . . . . . . . . . . . . 256
5.3.1 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 259
5.3.2 Selection. . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
5.3.3 Sorting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
5.3.4 Allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
5.3.5 Initialization Phase . . . . . . . . . . . . . . . . . . . . . . 265
5.4 SimE Operators and Parameters . . . . . . . . . . . . . . . . . . 266
5.4.1 Effect of Selection . . . . . . . . . . . . . . . . . . . . . . 267
5.4.2 Effect of Allocation . . . . . . . . . . . . . . . . . . . . . 267
5.4.3 Effect of Number of Populations . . . . . . . . . . . . . . 268
5.4.4 Effect of Number of Iterations . . . . . . . . . . . . . . . 268
5.5 Comparison of SimE, SA, and GA . . . . . . . . . . . . . . . . . 268
5.6 SimE Convergence Aspects . . . . . . . . . . . . . . . . . . . . . 269
5.7 SimE Applications . . . . . . . . . . . . . . . . . . . . . . . . . . 273
5.7.1 Standard-Cell Design Style . . . . . . . . . . . . . . . . . 273
5.7.2 Other Applications of SimE . . . . . . . . . . . . . . . . 284
8 CONTENTS
5.8 Parallelization of SimE . . . . . . . . . . . . . . . . . . . . . . . 285
5.8.1 Implementation of SimE on Vector Processors . . . . . . . 285
5.8.2 Implementation of SimE on MISD and MIMD machines . 287
5.9 Conclusion and Recent Work . . . . . . . . . . . . . . . . . . . . 289
6 Stochastic Evolution (StocE) 299
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
6.2 Historical Background . . . . . . . . . . . . . . . . . . . . . . . . 300
6.3 Stochastic Evolution Algorithm . . . . . . . . . . . . . . . . . . . 301
6.3.1 StocE Algorithm . . . . . . . . . . . . . . . . . . . . . . . 304
6.3.2 StocE versus SA . . . . . . . . . . . . . . . . . . . . . . . 307
6.3.3 StocE versus SimE . . . . . . . . . . . . . . . . . . . . . 308
6.4 StocE Convergence Aspects . . . . . . . . . . . . . . . . . . . . . 310
6.5 StocE Applications . . . . . . . . . . . . . . . . . . . . . . . . . . 312
6.5.1 Network Bisection . . . . . . . . . . . . . . . . . . . . . . 312
6.5.2 Vertex Cover Problem . . . . . . . . . . . . . . . . . . . . 313
6.5.3 Hamiltonian Circuit Problem . . . . . . . . . . . . . . . . 315
6.5.4 Traveling Salesman Problem . . . . . . . . . . . . . . . . 319
6.5.5 StocE based Technology Mapping of FPGAs . . . . . . . 320
6.6 Parallelization of StocE . . . . . . . . . . . . . . . . . . . . . . . 331
6.7 Conclusion and Recent Work . . . . . . . . . . . . . . . . . . . . 337
7 Hybrids and Other Issues 347
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347
7.2 Overview of Algorithms . . . . . . . . . . . . . . . . . . . . . . . 348
7.3 Hybridization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350
7.3.1 SA/TS Hybrid . . . . . . . . . . . . . . . . . . . . . . . . 350
7.3.2 GA/SA Hybrid . . . . . . . . . . . . . . . . . . . . . . . . 351
7.3.3 Others hybrids . . . . . . . . . . . . . . . . . . . . . . . . 353
7.4 GA and Multi-objective Optimization . . . . . . . . . . . . . . . 354
7.4.1 Pareto Optimality . . . . . . . . . . . . . . . . . . . . . . 355
7.4.2 VEGA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356
7.4.3 MOGA . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356
7.5 Fuzzy Logic for Multi-objective Optimization . . . . . . . . . . . 358
7.5.1 Fuzzy Set Theory. . . . . . . . . . . . . . . . . . . . . . . 358
7.5.2 Fuzzy Operators . . . . . . . . . . . . . . . . . . . . . . . 360
7.5.3 Fuzzy Rules . . . . . . . . . . . . . . . . . . . . . . . . . . 361
7.5.4 Example of Fuzzy Multi-objective Optimization . . . . . . 361
7.5.5 Fuzzy Goal Directed Optimization . . . . . . . . . . . . . 363
7.6 Artificial Neural Networks . . . . . . . . . . . . . . . . . . . . . 365
7.7 Solution Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . 369
7.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
Preface
Motivation
Combinatorial optimization problems are encountered everywhere, in science,
engineering, as well as in industrial management, economics, etc. Most
Engineering and business schools offer several courses in algorithms and
optimization. The advent of the digital computer is credited for all of this
explosion in the amount of algorithmic solutions to combinatorial optimization
problems. Such solution techniques were unthinkable before this magnificent
invention.
This book in concerned with one class of combinatorial optimization
algorithms: general iterative non-deterministic algorithms. The growing
interest in this class of algorithms is attributed to their generality, ease of
implementation, and mainly, the many success stories reporting very positive
results. We shall limit ourselves to five dominant iterative non-deterministic
algorithms, which, in order of popularity are: (1) Simulated Annealing (SA),
(2) Genetic Algorithm (GA), (3) Tabu Search (TS), (4) Simulated Evolution,
and (5) Stochastic Evolution. All five search heuristics have several important
properties in common.
1. They are blind, in that they do not know when they reached the optimal
solution. Therefore they must be told when to stop.
2. Theyareapproximationalgorithms,thatis,theydonotguaranteefinding
an optimal solution.
3. Theyhave‘hillclimbing’property, thatis, theyoccasionallyacceptuphill
(bad) moves.
4. They are easy to implement. All that is required is to have a suitable
solution representation, a cost function, and a mechanism to traverse the
search space.
5. They are all ‘general’. Practically they can be applied to solve any
combinatorial optimization problem.
9
10 CONTENTS
6. They all strive to exploit domain specific heuristic knowledge to bias the
searchtoward“good”solutionsubspace. Thequalityofsubspacesearched
depends to a large extent on the amount of heuristic knowledge used.
7. Although they asymptotically converge to an optimal solution, the rate
of convergence is heavily dependent on the adequate choice of several
parameters.
The last two properties are the hidden bone in the five combinatorial
optimization strategies. Our main goal in this book is to address this aspect by
compiling in a single source the extensive research work related to the problem
of intelligently setting the required parameters of these five heuristics.
Mostbooksoncomputeralgorithmsmainlyaddressdeterministicheuristics.
Recently, due to the increase in size and complexity of a large number
of combinatorial optimization problems, there has been a growing interest
in general iterative non-deterministic algorithms. There are several books
dedicated to one particular iterative algorithm. For example, there are at least
three books that introduce the theory and concepts of simulated annealing.
They are rigorous in mathematics and go far beyond the level of concepts
required by engineers (and scientists). There is at least one excellent text
book that introduces the theory and concepts of genetic algorithms. The
field of genetic algorithms is relatively new and the technique has only been
recently attempted to solve several NP-hard problem. Other non-deterministic
techniquessuchastabusearch,simulatedevolution,andstochasticevolutionare
now gaining ground and applications of such techniques have begun to appear
inscientificliterature. Tabusearchhasrecentlybeenthesubjectofanexcellent
bookbyF.Glover(KluwerPublishers,1997). Currently,toourknowledgethere
arenobooksthataddressthelasttwotechniques. And,nobookisavailablethat
contains an integrated and up-to-date description of all the above techniques,
with case studies and examples.
Allfiveheuristicsdescribedinthisbookconstituteverygeneralandeffective
optimizationtechniques. Recently,SA,GA,andTShasbeendesignatedbythe
CommitteeoftheNextDecadeofOperationsResearchas‘extremelypromising’
forthefuturetreatmentofpracticalapplications1. ItisourbeliefthatSimulated
Evolution and Stochastic Evolution are equally promising techniques to a wide
array of combinatorial optimization problems.
Organization of the Book
The book is organized into 7 chapters. The introductory chapter motivates
the student towards a study of iterative algorithms. The definition of NP-hard
and NP-complete is introduced, and some “hard” problems are illustrated with
examples.
1F. Glover, E. Taillard, and D. de Werra. A user’s guide to tabu search. Annals of
Operations Research,41:3–28,1993.