Algorithms Jeff Erickson (UIUC CS473 573)

514 Pages·2017·10.626 MB·English

by  it-ebooks

Algorithms
Jeff Erickson
January4,2015
http://www.cs.illinois.edu/~jeffe/teaching/algorithms/
©Copyright1999–2015JeffErickson.LastupdateJanuary4,2015.
Thisworkmaybefreelycopiedanddistributedinanymedium.
Itmaynotbesoldformorethantheactualcostofreproduction,storage,ortransmittal.
ThisworkisavailableunderaCreativeCommonsAttribution-NonCommercial-ShareAlike4.0InternationalLicense.
Forlicensedetails,seehttp://creativecommons.org/licenses/by-nc-sa/4.0/.
Forthemostrecentedition,seehttp://www.cs.illinois.edu/~jeffe/teaching/algorithms/.
ShallItellyou,myfriend,howyouwillcometounderstandit?
Goandwriteabookonit.
—HenryHome,LordKames(1696–1782),toSirGilbertElliot
Theindividualisalwaysmistaken.Hedesignedmanythings,anddrewinother
personsascoadjutors,quarrelledwithsomeorall,blunderedmuch,andsome-
thingisdone;allarealittleadvanced,buttheindividualisalwaysmistaken. It
turnsoutsomewhatnewandveryunlikewhathepromisedhimself.
—RalphWaldoEmerson,“Experience”,Essays,SecondSeries(1844)
Theoreticallecturesshouldneitherbeareproductionof
noracommentuponanytext-book,howeversatisfactory.
Thestudent’snotebookshouldbehisprincipaltext-book.
—AndréWeil,“MathematicalTeachinginUniversities”(1954)
About These Notes
These are lecture notes that I wrote for various algorithms classes at the University of
Illinois at Urbana-Champaign, which I have taught on average once a year since January
1999. The most recent revision of these notes (or nearly so) is available online at http:
//www.cs.illinois.edu/~jeffe/teaching/algorithms/,alongwithanear-completearchiveofall
my past homeworks and exams. Whenever I teach an algorithms class, I revise, update, and
sometimescullthesenotesasthecourseprogresses,soyoumayfindmorerecentversionsonthe
webpageofwhatevercourseIamcurrentlyteaching.
Withfewexceptions,eachofthese“lecturenotes”containsfartoomuchmaterialtocoverin
a single lecture. In a typical 75-minute class period, I cover about 4 or 5 pages of material—a
bitmoreifI’mteachinggraduatestudentsthanundergraduates. Moreover,Icanonlycoverat
mosttwo-thirdsofthesenotesinany capacityinasingle15-weeksemester. Yourmileagemay
vary! (Arguably,thatmeansthatasIcontinuetoaddmaterial,thelabel“lecturenotes”becomes
lessandlessaccurate.) Iteachalgorithmsatmultipleleaves;differentcoursescoverdifferent
butoverlappingsubsetsofthismaterial. Theorderingofthenotesismostlyconsistentwithmy
lower-levelclasses,withmoreadvancedmaterial(indicatedby stars)insertednearthemore
∗
basicmaterialitbuildson. Theactualmaterialdoesn’tpermitastrictlinearordering,butI’ve
triedtokeepforwardreferencestoaminimum.
About the Exercises
Each note ends with several exercises, most of which have been used at least once in a
homeworkassignment,discussionsection,orexam. (cid:63)Starsindicatemorechallengingproblems;
manyofthesestarredproblemsappearedonqualifyingexamsforthealgorithmsPhDstudents
at UIUC. A small number of really hard problems are marked with a (cid:198)larger star; one or two
openproblemsareindicatedby enormousstars. Manyoftheseexerciseswerecontributedby
(cid:198)
myamazingteachingassistants:
Aditya Ramani, Akash Gautam, Alex Steiger, Alina Ene, Amir Nayyeri, Asha
Seetharam, Ashish Vulimiri, Ben Moseley, Brad Sturt, Brian Ensink, Chao Xu,
ChrisNeihengen,ConnorClark,DanBullok,DanCranston,DanielKhashabi,David
Morrison,JohnathonFischer,JunqingDeng,EktaManaktala,ErinWolfChambers,
i
GailSteitz,GioKao,GrantCzajkowski,Hsien-ChihChang,IgorGammer,JohnLee,
Kent Quanrud, Kevin Milans, Kevin Small, Kyle Fox, Kyle Jao, Lan Chen, Michael
Bond,MitchHarris,NaveenArivazhagen,NickBachmair,NickHurlburt,NirmanKu-
mar,NitishKorula,RachitAgarwal,RezaZamani-Nasab,RishiTalreja,RobMcCann,
ShripadThite,SubhroRoy,TanaWattanawaroon,andYasuFurakawa.
Please do not ask me for solutions to the exercises. If you are a student, seeing the
solutionwillrobyouoftheexperienceofsolvingtheproblemyourself,whichistheonlywayto
learnthematerial. Ifyouareaninstructor,youshouldn’tassignproblemsthatyoucan’tsolve
yourself! (BecauseIdon’talwaysfollowmyownadvice,Isometimesassignbuggyproblems,but
I’vetriedtokeeptheseoutofthelecturenotesthemselves.)
“Johnny’s”multi-coloredcrayonhomeworkwasfoundundertheTAofficedooramongthe
otherFall2000Homework1submissions.
Acknowledgments
All of this material draws heavily on the creativity, wisdom, and experience of thousands
of algorithms students, teachers, and researchers. In particular, I am immensely grateful to
themorethan2000Illinoisstudentswhohaveusedthesenotesasaprimaryreference,offered
useful(ifsometimespainful)criticism,andsufferedthroughsometrulyawfulfirstdrafts. I’m
alsogratefulforthecontributionsandfeedbackfromtheteachingassistantslistedabove. Finally,
thanks to many colleagues at Illinois and elsewhere who have used these notes in their own
classesandhavesenthelpfulfeedbackandbugreports.
Naturally,thesenotesoweagreatdealtothepeoplewhotaughtmethisalgorithmsstuffin
thefirstplace: BobBixbyandMichaelPearlmanatRice; DavidEppstein, DanHirshberg,and
George Lueker at UC Irvine; and Abhiram Ranade, Dick Karp, Manuel Blum, Mike Luby, and
RaimundSeidelatUCBerkeley. I’vealsobeenhelpedtremendouslybymanydiscussionswith
facultycolleaguesatIllinois—CindaHeeren,EdgarRamos,HerbertEdelsbrunner,JasonZych,
LennyPitt,MadhuParasarathy,MaheshViswanathan,MargaretFleck,Shang-HuaTeng,Steve
LaValle, and especially Chandra Chekuri, Ed Reingold, and Sariel Har-Peled. I stole the first
iteration of the overall course structure, and the idea to write up my own lecture notes, from
HerbertEdelsbrunner.
The picture of the Spirit of Arithmetic from Margarita Philosophica at the end of the
introductorynoteswascopiedfromWikimediaCommons;theoriginal1508woodcutisinthe
public domain. The map on the first page of the maxflow/mincut notes was copied from Lex
Schrijver’sexcellentsurvey“Onthehistoryofcombinatorialoptimization(till1960)”;theoriginal
map is from a US Government work in the public domain. Several of Randall Munroe’s xkcd
comicstripsarereproducedunderaCreativeCommonsLicense. Onewell-knownframefrom
AllieBrosh’scomicstripHyperboleandaHalf appearstwicewithoutpermission. (Hireallthe
lawyers?)
I drew all other figures in the notes myself using OmniGraffle, except for a few older
figures that I drew with (shudder) xfig. In particular, the square-Kufi rendition of the name
“al-Khwa¯rizm¯ı”onthecoverismyown.
ii
Prerequisites
These notes assume the reader has mastered the material covered in the first two years
of a strong undergraduate computer science curriculum, and that they have the intellectual
maturity to recognize and repair any remaining gaps in their mastery. In particular, for most
students,thesenotesarenotsuitableforafirstcourseindatastructuresandalgorithms. Specific
prerequisitesincludethefollowing:
• Discretemathematics: High-schoolalgebra,logarithmidentities,naivesettheory,Boolean
algebra,first-orderpredicatelogic,sets,functions,equivalences,partialorders,modular
arithmetic,recursivedefinitions,trees(asabstractobjects,notdatastructures),graphs.
• Proof techniques: direct,indirect,contradiction,exhaustivecaseanalysis,andinduction
(especially“strong”and“structural”induction). Lecture0requiresinduction,andwhenever
Lecture n 1requiresinduction,sodoesLecture n.
−
• Elementary discrete probability: uniform vs non-uniform distributions, expectation,
conditionalprobability,linearityofexpectation,independence.
• Iterative programming concepts: variables,conditionals,loops,indirection(addresses/
pointers/references), subroutines, recursion. I do not assume fluency in any particular
programming language, but I do assume experience with at least one language that
supportsindirectionandrecursion.
• Fundamental abstract data types: scalars, sequences, vectors, sets, stacks, queues,
priorityqueues,dictionaries.
• Fundamentaldatastructures: arrays,linkedlists(singleanddouble,linearandcircular),
binary search trees, at least one balanced binary search tree (AVL trees, red-black trees,
treaps, skip lists, splay trees, etc.), binary heaps, hash tables, and most importantly, the
differencebetweenthislistandthepreviouslist.
• Fundamental algorithmic problems: sorting,searching,enumeration.
• Fundamental algorithms: elementary arithmetic, sequential search, binary search,
comparison-based sorting (selection, insertion, merge-, heap-, quick-), radix sort, pre-
/post-/inordertreetraversal,breadth-anddepth-firstsearch(atleastintrees),andmost
importantly,thedifferencebetweenthislistandthepreviouslist.
• Basic algorithm analysis: Asymptotic notation (o, O, Θ, Ω, ω), translating loops into
sumsandrecursivecallsintorecurrences,evaluatingsimplesumsandrecurrences.
• Mathematical maturity: facility with abstraction, formal (especially recursive) defini-
tions,and(especiallyinductive)proofs;writingandfollowingmathematicalarguments;
recognizingandavoidingsyntactic,semantic,and/orlogicalnonsense.
Two notes on prerequisite material appear as an appendix to the main lecture notes: one
on proofs by induction, and one on solving recurrences. The main lecture notes also briefly
coversomeprerequisitematerial,butmoreasareminderthanagoodintroduction. Foramore
thoroughoverview,Istronglyrecommendthefollowing:
• MargaretM.Fleck. BuildingBlocksforTheoreticalComputerScience,unpublishedtextbook,
mostrecentlyrevisedJanuary2013.
• EricLehman,F.ThomsonLeighton,andAlbertR.Meyer. MathematicsforComputerScience,
unpublishedlecturenotes,mostrecent(public)revisionJanuary2013.
iii
• Pat Morin. Open Data Structures, most recently revised June 2014 (edition 0.1G). A
permanentlyfreeopen-sourcetextbook,whichPatmaintainsandregularlyupdates.
Additional References
I strongly encourage students (and other readers) not to restrict themselves to my notes
oranyothersingletextualreference. Authorsandreadersbringtheirownperspectivestothe
material;noinstructor“clicks”witheverystudent,oreveneveryverystrongstudent. Findingthe
authorthatmosteffectivelygetstheirintuitionintoyourheadtakesomeeffort,butthateffort
paysoffhandsomelyinthelongrun. Thefollowingreferenceshavebeenparticularlyvaluableto
meassourcesofinspiration,intuition,examples,andproblems.
• AlfredV.Aho,JohnE.Hopcroft,andJeffreyD.Ullman. TheDesignandAnalysisofComputer
Algorithms. Addison-Wesley,1974. (IusedthistextbookasanundergraduateatRiceand
againasamastersstudentatUCIrvine.)
• ThomasCormen,CharlesLeiserson,RonRivest,andCliffStein. IntroductiontoAlgorithms,
thirdedition. MITPress/McGraw-Hill,2009. (Iusedthefirsteditionasateachingassistant
atBerkeley.)
• SanjoyDasgupta,ChristosH.Papadimitriou,andUmeshV.Vazirani. Algorithms. McGraw-
Hill,2006.
• JeffEdmonds. HowtoThinkaboutAlgorithms. CambridgeUniversityPress,2008.
• Michael R. Garey and David S. Johnson. Computers and Intractability: A Guide to the
TheoryofNP-Completeness. W.H.Freeman,1979.
• MichaelT.GoodrichandRobertoTamassia. AlgorithmDesign: Foundations,Analysis,and
InternetExamples. JohnWiley&Sons,2002.
• JohnE.HopcroftandJeffreyD.Ullman. IntroductiontoAutomataTheory,Languages,and
Computation,firstedition. Addison-Wesley,1979. (Iusedthistextbookasanundergraduate
atRice. Don’tbotherwiththelatereditions.)
• JonKleinbergandÉvaTardos. AlgorithmDesign. Addison-Wesley,2005.
• DonaldKnuth. TheArtofComputerProgramming,volumes1–4A.Addison-Wesley,1997and
2011. (MyparentsgavemethefirstthreevolumesforChristmaswhenIwas14,butIdidn’t
actuallyreadthemuntilmuchlater.)
• UdiManber. IntroductiontoAlgorithms: ACreativeApproach. Addison-Wesley,1989. (Iused
thistextbookasateachingassistantatBerkeley.)
• RajeevMotwaniandPrabhakarRaghavan. RandomizedAlgorithms. CambridgeUniversity
Press,1995.
• Ian Parberry. Problems on Algorithms. Prentice-Hall, 1995 (out of print). Available from
http://www.eng.unt.edu/ian/books/free/license.html after promising to make a small
charitabledonation. Pleasehonoryourpromise.
• AlexanderSchrijver. CombinatorialOptimization: PolyhedraandEfficiency. Springer,2003.
(JustincaseyouthoughtKnuthwastheonlyauthorwhocouldstunoxen.)
• RobertSedgewickandKevinWayne. Algorithms. Addison-Wesley,2011.
iv
• JeffreyO.Shallit. ASecondCourseinFormalLanguagesandAutomataTheory. Cambridge
UniversityPress,2008.
• MichaelSipser. IntroductiontotheTheoryofComputation,thirdedition. CengageLearning,
2012. Recommendedifandonlyifyoudon’thavetopayforit.
• RobertEndreTarjan. DataStructuresandNetworkAlgorithms. SIAM,1983.
• RobertJ.Vanderbei. LinearProgramming: FoundationsandExtensions. Springer,2001.
• ClassnotesfrommyownalgorithmsclassesatBerkeley,especiallythosetaughtbyDick
KarpandRaimundSeidel.
• Lecturenotes,slides,homeworks,exams,videolectures,researchpapers,blogposts,and
full-fledgedMOOCsmadefreelyavailableonthewebbyinnumerablecolleaguesaround
theworld.
Caveat Lector!
Despite several rounds of revision, these notes still contain mnay mistakes, errors, bugs,
gaffes, omissions, snafus, kludges, typos, mathos, grammaros, thinkos, brain farts, nonsense,
garbage,cruft,junk,andoutrightlies,all of which are entirely Steve Skiena’s fault. Irevise
andupdatethesenoteseverytimeIteachanalgorithmsclass,sopleaseletmeknowifyoufinda
bug. (Steveisunlikelytocare.) Iregularlyawardextracredittostudentswhopostexplanations
and/orcorrectionsoferrorsinthelecturenotes. IfI’mnotteachingyourclass,encourageyour
instructortosetupasimilarextra-creditscheme,andforwardthebugreportstoSteveme!
Ofcourse,anyotherfeedbackisalsowelcome!
Enjoy!
—Jeff
Itistraditionalfortheauthortomagnanimouslyaccepttheblameforwhateverdeficien-
ciesremain.Idon’t.Anyerrors,deficiencies,orproblemsinthisbookaresomebody
else’sfault,butIwouldappreciateknowingaboutthemsoastodeterminewhoisto
blame.
—StevenS.Skiena,TheAlgorithmDesignManual(1997)
v
vi
Contents
0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
I Recursion 17
1 Recursion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2 (cid:63)FastFourierTransforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3 Backtracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4 (cid:63)EfficientExponential-TimeAlgorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5 DynamicProgramming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
6 (cid:63)AdvancedDynamicProgramming. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
7 GreedyAlgorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
8 (cid:63)Matroids. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
II Randomization 139
9 RandomizedAlgorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
10 RandomizedBinarySearchTrees. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
11 (cid:63)TailInequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
12 Hashing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
13 StringMatching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
14 RandomizedMinimumCut . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
III Amortization 215
15 AmortizedAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
16 ScapegoatandSplayTrees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
17 DisjointSets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
IV Graphs 261
18 BasicGraphAlgorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
19 Depth-FirstSearch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
20 MinimumSpanningTrees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
21 Single-SourceShortestPaths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .303
22 All-PairsShortestPaths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
V Optimization 331
23 MaximumFlowsandMinimumCuts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .333
24 ApplicationsofMaximumFlow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347
25 (cid:63)ExtensionsofMaximumFlow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .359
vii
26 (cid:63)LinearProgramming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .369
27 (cid:63)TheSimplexAlgorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
VI Hardness 389
28 LowerBounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
29 AdversaryArguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397
30 NP-HardProblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .405
31 (cid:63)ApproximationAlgorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .435
VII Appendices 453
A ProofsbyInduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 455
B SolvingRecurrences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .483
viii