Algorithms - Mathematics & Computer Science

318 Pages·2012·1.93 MB·English

Algorithms
Copyright c 2006 S. Dasgupta, C. H. Papadimitriou, and U. V. Vazirani
(cid:13)
July 18, 2006
2
Contents
Preface 9
0 Prologue 11
0.1 Books andalgorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
0.2 Enter Fibonacci . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
0.3 Big-O notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
1 Algorithmswith numbers 21
1.1 Basicarithmetic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
1.2 Modulararithmetic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
1.3 Primalitytesting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
1.4 Cryptography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
1.5 Universalhashing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Randomizedalgorithms: a virtualchapter 38
2 Divide-and-conquer algorithms 51
2.1 Multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
2.2 Recurrence relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
2.3 Mergesort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
2.4 Medians. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
2.5 Matrixmultiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
2.6 Thefast Fourier transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
3 Decompositionsof graphs 87
3.1 Whygraphs? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
3.2 Depth-(cid:2)rst search inundirected graphs . . . . . . . . . . . . . . . . . . . . . . . . 89
3.3 Depth-(cid:2)rst search indirected graphs . . . . . . . . . . . . . . . . . . . . . . . . . . 94
3.4 Strongly connected components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
3
4 Paths in graphs 109
4.1 Distances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
4.2 Breadth-(cid:2)rst search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
4.3 Lengthsonedges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
4.4 Dijkstra’salgorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
4.5 Priority queueimplementations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
4.6 Shortest pathsinthe presence of negativeedges . . . . . . . . . . . . . . . . . . . 122
4.7 Shortest pathsindags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
5 Greedy algorithms 133
5.1 Minimumspanningtrees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
5.2 Huffmanencoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
5.3 Horn formulas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
5.4 Set cover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
6 Dynamicprogramming 161
6.1 Shortest pathsindags,revisited . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
6.2 Longest increasingsubsequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
6.3 Edit distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
6.4 Knapsack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
6.5 Chainmatrixmultiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
6.6 Shortest paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
6.7 Independent sets intrees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
7 Linear programmingand reductions 189
7.1 Anintroduction to linearprogramming . . . . . . . . . . . . . . . . . . . . . . . . 189
7.2 Flowsinnetworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
7.3 Bipartite matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
7.4 Duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
7.5 Zero-sum games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
7.6 Thesimplexalgorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
7.7 Postscript: circuit evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
8 NP-completeproblems 233
8.1 Search problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
8.2 NP-complete problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
8.3 Thereductions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
4
9 CopingwithNP-completeness 269
9.1 Intelligentexhaustivesearch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
9.2 Approximationalgorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
9.3 Localsearch heuristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
10 Quantum algorithms 295
10.1 Qubits,superposition, andmeasurement . . . . . . . . . . . . . . . . . . . . . . . 295
10.2 Theplan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
10.3 ThequantumFouriertransform . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
10.4 Periodicity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
10.5 Quantumcircuits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
10.6 Factoring asperiodicity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
10.7 Thequantumalgorithmfor factoring . . . . . . . . . . . . . . . . . . . . . . . . . . 308
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
Historicalnotes andfurther reading 313
Index 315
5
6
List of boxes
Basesandlogs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Two’scomplement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Is yoursocialsecurity numberaprime? . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Hey, thatwasgrouptheory! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Carmichaelnumbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Randomizedalgorithms: avirtualchapter . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Anapplicationof numbertheory? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Binarysearch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Annlognlower boundfor sorting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
TheUnixsortcommand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Whymultiplypolynomials? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Theslowspread of afast algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
How bigisyourgraph? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Crawlingfast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
Whichheapisbest? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
A randomizedalgorithmfor minimumcut . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
Recursion? No, thanks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
Programming? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
Commonsubproblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
Of miceandmen. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
Memoization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
Ontimeandmemory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
A magictrick calledduality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Reductions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
Matrix-vector notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
Visualizingduality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
Gaussianelimination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
Linearprogramminginpolynomialtime . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
Thestory of SissaandMoore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
7
WhyPandNP? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
Thetwo waysto usereductions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Unsolvableproblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
Entanglement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
TheFourier transform of aperiodic vector . . . . . . . . . . . . . . . . . . . . . . . . . . 303
Setting upaperiodic superposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
Quantumphysicsmeets computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
8
Preface
Thisbookevolvedoverthepasttenyearsfromasetoflecturenotesdeveloped whileteaching
the undergraduate Algorithms course at Berkeley and U.C. San Diego. Our way of teaching
thiscourseevolvedtremendouslyovertheseyearsinanumberofdirections,partlytoaddress
our students’ background (undeveloped formal skills outside of programming), and partly to
re(cid:3)ect the maturing of the (cid:2)eld in general, as we have come to see it. The notes increasingly
crystallized into a narrative, and we progressively structured the course to emphasize the
(cid:147)story line(cid:148) implicit in the progression of the material. As a result, the topics were carefully
selected and clustered. No attempt was made to be encyclopedic, and this freed us to include
topics traditionallyde-emphasized or omitted from most Algorithmsbooks.
Playing on the strengths of our students (shared by most of today’s undergraduates in
Computer Science), instead of dwelling on formal proofs we distilled in each case the crisp
mathematicalideathatmakesthealgorithmwork. Inotherwords,weemphasizedrigorover
formalism. We found that our students were much more receptive to mathematical rigor of
thisform. It isthisprogression of crisp ideasthathelpsweave the story.
Once you think about Algorithms in this way, it makes sense to start at the historical be-
ginning of it all, where, in addition, the characters are familiar and the contrasts dramatic:
numbers, primality, and factoring. This is the subject of Part I of the book, which also in-
cludes the RSA cryptosystem, and divide-and-conquer algorithms for integer multiplication,
sortingandmedian(cid:2)nding,aswellasthefastFouriertransform. Therearethreeotherparts:
Part II, the most traditional section of the book, concentrates on data structures and graphs;
the contrast here isbetween the intricate structure of the underlyingproblems andthe short
and crisp pieces of pseudocode that solve them. Instructors wishing to teach a more tradi-
tional course can simply start with Part II, which is self-contained (following the prologue),
and then cover Part I as required. In Parts I and II we introduced certain techniques (such
as greedy and divide-and-conquer) which work for special kinds of problems; Part III deals
with the (cid:147)sledgehammers(cid:148) of the trade, techniques that are powerful and general: dynamic
programming (a novel approach helps clarify this traditional stumbling block for students)
and linear programming (a clean and intuitive treatment of the simplex algorithm, duality,
and reductions to the basic problem). The (cid:2)nal Part IV is about ways of dealing with hard
problems: NP-completeness, various heuristics, as well as quantum algorithms, perhaps the
most advanced and modern topic. As it happens, we end the story exactly where we started
it, withShor’squantumalgorithmfor factoring.
The book includes three additional undercurrents, in the form of three series of separate
(cid:147)boxes,(cid:148) strengthening the narrative (and addressing variationsin the needs and interests of
thestudents)whilekeepingthe(cid:3)owintact: piecesthatprovidehistoricalcontext;descriptions
ofhowtheexplainedalgorithmsareusedinpractice(withemphasisoninternetapplications);
9
andexcursionsfor themathematicallysophisticated.
10