Table Of ContentA n I n t r o d u c t i o n t o t h e
Analysis of Algorithms
2nd Edition
8439hc.9789814401159-tp.indd 1 25/5/12 5:35 PM
TThhiiss ppaaggee iinntteennttiioonnaallllyy lleefftt bbllaannkk
A n I n t r o d u c t i o n t o t h e
Analysis of Algorithms
2nd Edition
Michael Soltys
McMaster University, Canada
World Scientific
NEW JERSEY • LONDON • SINGAPORE • BEIJING • SHANGHAI • HONG KONG • TAIPEI • CHENNAI
8439hc.9789814401159-tp.indd 2 25/5/12 5:35 PM
Published by
World Scientific Publishing Co. Pte. Ltd.
5 Toh Tuck Link, Singapore 596224
USA office: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601
UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE
British Library Cataloguing-in-Publication Data
A catalogue record for this book is available from the British Library.
AN INTRODUCTION TO THE ANALYSIS OF ALGORITHMS
2nd Edition
Copyright © 2012 by World Scientific Publishing Co. Pte. Ltd.
All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means,
electronic or mechanical, including photocopying, recording or any information storage and retrieval
system now known or to be invented, without written permission from the Publisher.
For photocopying of material in this volume, please pay a copying fee through the Copyright
Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to
photocopy is not required from the publisher.
ISBN-13 978-981-4401-15-9
ISBN-10 981-4401-15-3
Printed in Singapore.
Patrick - An Intro to the Analysis (2nd Ed).pmd1 3/20/2012, 7:32 PM
April3,2012 10:24 WorldScientificBook-9inx6in soltys˙alg
To my family
v
April3,2012 10:24 WorldScientificBook-9inx6in soltys˙alg
TThhiiss ppaaggee iinntteennttiioonnaallllyy lleefftt bbllaannkk
April3,2012 10:24 WorldScientificBook-9inx6in soltys˙alg
Preface
Thisbookisanintroductiontotheanalysisofalgorithms,fromthepointof
view of proving algorithm correctness. Our theme is the following: how do
we argue mathematically, without a burden of excessive formalism, that a
givenalgorithmdoeswhatitissupposedtodo? Andwhyisthisimportant?
In the words of C.A.R. Hoare:
Asfarasthefundamentalscienceisconcerned,westillcertainly
do not know how to prove programs correct. We need a lot of
steadyprogressinthisarea,whichonecanforesee,andalotof
breakthroughswherepeoplesuddenlyfindthere’sasimpleway
todosomethingthateverybodyhithertohasthoughttobefar
too difficult1.
Software engineers know many examples of things going terribly wrong
becauseofprogramerrors; theirparticularfavoritesarethefollowingtwo2.
The blackout in the American North-East during the summer of 2003 was
due to a software bug in an energy management system; an alarm that
shouldhavebeentriggeredneverwentoff, leadingtoachainofeventsthat
climaxed in a cascading blackout. The Ariane 5, flight 501, the maiden
flight of the rocket in June 4, 1996, ended with an explosion 40 seconds
into the flight; this $500 million loss was caused by an overflow in the
conversion from a 64-bit floating point number to a 16-bit signed integer.
While the goal of absolute certainty in program correctness is elusive,
we can develop methods and techniques for reducing errors. The aim of
this book is modest: we want to present an introduction to the analysis
of algorithms—the “ideas” behind programs, and show how to prove their
correctness.
1FromAn Interview with C.A.R. Hoare,in[Shustek(2009)].
2These two examples come from [van Vliet (2000)], where many more instances of
spectacularfailuresmaybefound.
vii
April3,2012 10:24 WorldScientificBook-9inx6in soltys˙alg
viii An Introduction to the Analysis of Algorithms
The algorithm may be correct, but the implementation itself might be
flawed. Some syntactical errors in the program implementation may be
uncoveredbyacompilerortranslator—whichinturncouldalsobebuggy—
but there might be other hidden errors. The hardware itself might be
faulty; the libraries on which the program relies at run time might be
unreliable, etc. It is the task of the software engineer to write code that
works given such a delicate environment, prone to errors. Finally, the
algorithmiccontentofapieceofsoftwaremightbeverysmall;themajority
of the lines of code could be the “menial” task of interface programming.
Thus, the ability to argue correctly about the soundness of an algorithm is
only one of many facets of the task at hand, yet an important one, if only
forthepedagogicalreasonoflearningtoarguerigorouslyaboutalgorithms.
We begin this book with a chapter of preliminaries, containing the key
ideasofinductionandinvariance,andtheframeworkofpre/post-conditions
and loop invariants. We also prove the correctness of some classical algo-
rithms, such as the integer division algorithm, and Euclid’s procedure for
computing the greatest common divisor of two numbers.
We present three standard algorithm design techniques in eponymous
chapters: greedy algorithms, dynamic programming and the divide and
conquerparadigm. Weareconcernedwithcorrectnessofalgorithms,rather
than, say, efficiency or the underlying data structures. For example, in the
chapteronthegreedyparadigmweexploreindepththeideaofapromising
partial solution, a powerful technique for proving the correctness of greedy
algorithms. We include online algorithms and competitive analysis, as well
as randomized algorithms with a section on cryptography.
Algorithms solve problems, and many of the problems in this book fall
under the category of optimization problems, whether cost minimization,
such as Kruskal’s algorithm for computing minimum cost spanning trees—
section 2.1, or profit maximization, such as selecting the most profitable
subset of activities—section 4.4.
Thebookissprinkledwithproblems;mosttesttheunderstandingofthe
material, but many include coding in the Python programming language.
The reader is expected to learn Python on their own (a great source to do
sois[Downey(2008)]—thePDFcanbedownloadedforfreefromtheweb).
One of the advantages of this programming language is that it is easy to
start writing small snippets of code that work—and most of the coding in
this book falls into the “small snippet” category. The solutions to most
problems are included in the “Answers to selected problems” at the end of
each chapter. The solutions to most of the programming exercises will be
April3,2012 10:24 WorldScientificBook-9inx6in soltys˙alg
Preface ix
available for download from the author’s web page (a quick Google search
will reveal the URL).
Theintendedaudienceforthisbookconsistsofundergraduatestudents
incomputerscienceandmathematics. Thebookisveryself-contained: the
first chapter, Preliminaries, reviews induction and the invariance principle.
It also introduces the aforementioned ideas of pre/post-conditions, loop
invariants and termination—in other words, it sets the mathematical stage
for the rest of the book. Not much mathematics is assumed (besides some
tameforaysintolinearalgebraandnumbertheory),butacertainpenchant
for discrete mathematics is considered helpful.
This book draws on many sources. First of all, [Cormen et al. (2009)]
is a fantastic reference for anyone who is learning algorithms. I have also
used as reference the elegantly written [Kleinberg and Tardos (2006)]. A
classic in the field is [Knuth (1997)], and I base my presentation of online
algorithmsonthematerialin[BorodinandEl-Yaniv(1998)]. Ihavelearned
greedy algorithms, dynamic programming and logic from Stephen A. Cook
attheUniversityofToronto. AppendixB,adigestofrelations,isbasedon
hand-written lecture slides of Ryszard Janicki.
No book on algorithms is complete without a short introduction to
the “big-Oh” notation. Consider functions from N to N. We say that
g(n) ∈ O(f(n)) if there exist constants c,n such that for all n ≥ n ,
0 0
g(n) ≤ cf(n), and the little-oh notation, g(n) ∈ o(f(n)), which denotes
thatlim g(n)/f(n)=0. Wealsosaythatg(n)∈Ω(f(n))ifthereexist
n→∞
constants c,n such that for all n≥n , g(n)≥cf(n). Finally, we say that
0 0
g(n)∈Θ(f(n)) if it is the case that g(n)∈O(f(n))∩Ω(f(n)).
The ubiquitous floor and ceil functions are defined, respectively, as fol-
lows: (cid:98)x(cid:99)=max{n∈Z|n≤x} and (cid:100)x(cid:101)=min{n∈Z|n≥x}. Finally, (cid:98)x(cid:101)
refers to the “rounding-up” of x, and it is defined as (cid:98)x(cid:101)=(cid:98)x+ 1(cid:99).
2
We have the usual Boolean connectives: ∧ is “and,” ∨ is “or” and ¬
is “not.” We also use → as Boolean implication, i.e., x → y is logically
equivalent to ¬x∨y, and ↔ is Boolean equivalence, and α ↔ β expresses
((α→β)∧(β →α))
We use “⇒” to abbreviate the word “implies,” i.e., 2|x ⇒ x is even,
while “(cid:54)⇒” abbreviates “does not imply.”
