From Algorithms to Z-Scores: Probabilistic and Statistical Modeling in Computer Science

565 Pages·2014·8.78 MB·English

From Algorithms to Z-Scores:
Probabilistic and Statistical Modeling in
Computer Science
Norm Matloff, University of California, Davis
library(MASS)
f (t) = ce−0.5(t−µ)(cid:48)Σ−1(t−µ) x <- mvrnorm(mu,sgm)
X
0.015
z0.010
0.005
10
5
−10 0
−5 x 2
0 −5
x1 5
10−10
See Creative Commons license at
http://heather.cs.ucdavis.edu/ matloff/probstatbook.html
The author has striven to minimize the number of errors, but no guarantee is made as to accuracy
of the contents of this book.
2
Author’s Biographical Sketch
Dr. Norm Matloff is a professor of computer science at the University of California at Davis, and
was formerly a professor of statistics at that university. He is a former database software developer
in Silicon Valley, and has been a statistical consultant for firms such as the Kaiser Permanente
Health Plan.
Dr. Matloff was born in Los Angeles, and grew up in East Los Angeles and the San Gabriel Valley.
HehasaPhDinpuremathematicsfromUCLA,specializinginprobabilitytheoryandstatistics. He
has published numerous papers in computer science and statistics, with current research interests
in parallel processing, statistical computing, and regression methodology.
Prof. Matloff is a former appointed member of IFIP Working Group 11.3, an international com-
mittee concerned with database software security, established under UNESCO. He was a founding
member of the UC Davis Department of Statistics, and participated in the formation of the UCD
Computer Science Department as well. He is a recipient of the campuswide Distinguished Teaching
Award and Distinguished Public Service Award at UC Davis.
Dr. Matloffistheauthoroftwopublishedtextbooks, andofanumberofwidely-usedWebtutorials
on computer topics, such as the Linux operating system and the Python programming language.
He and Dr. Peter Salzman are authors of The Art of Debugging with GDB, DDD, and Eclipse.
Prof. Matloff’s book on the R programming language, The Art of R Programming, was published
in 2011. His book, Parallel Computation for Data Science, will come out in 2014. He is also the
author of several open-source textbooks, including From Algorithms to Z-Scores: Probabilistic and
Statistical Modeling in Computer Science(http://heather.cs.ucdavis.edu/probstatbook),and
Programming on Parallel Machines (http://heather.cs.ucdavis.edu/~matloff/ParProcBook.
pdf).
Contents
1 Time Waste Versus Empowerment 1
2 Basic Probability Models 3
2.1 ALOHA Network Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 The Crucial Notion of a Repeatable Experiment . . . . . . . . . . . . . . . . . . . . 5
2.3 Our Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.4 “Mailing Tubes” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.5 Basic Probability Computations: ALOHA Network Example . . . . . . . . . . . . . 10
2.6 Bayes’ Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.7 ALOHA in the Notebook Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.8 Solution Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.9 Example: Divisibility of Random Integers . . . . . . . . . . . . . . . . . . . . . . . . 17
2.10 Example: A Simple Board Game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.11 Example: Bus Ridership . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.12 Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.12.1 Example: Rolling Dice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.12.2 Improving the Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.12.2.1 Simulation of Conditional Probability in Dice Problem . . . . . . . 24
2.12.3 Simulation of the ALOHA Example . . . . . . . . . . . . . . . . . . . . . . . 25
i
ii CONTENTS
2.12.4 Example: Bus Ridership, cont’d. . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.12.5 Back to the Board Game Example . . . . . . . . . . . . . . . . . . . . . . . . 27
2.12.6 How Long Should We Run the Simulation? . . . . . . . . . . . . . . . . . . . 27
2.13 Combinatorics-Based Probability Computation . . . . . . . . . . . . . . . . . . . . . 27
2.13.1 Which Is More Likely in Five Cards, One King or Two Hearts? . . . . . . . . 27
2.13.2 Example: Random Groups of Students . . . . . . . . . . . . . . . . . . . . . . 29
2.13.3 Example: Lottery Tickets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.13.4 “Association Rules” in Data Mining . . . . . . . . . . . . . . . . . . . . . . . 30
2.13.5 Multinomial Coefficients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.13.6 Example: Probability of Getting Four Aces in a Bridge Hand . . . . . . . . . 31
3 Discrete Random Variables 37
3.1 Random Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.2 Discrete Random Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.3 Independent Random Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.4 Expected Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.4.1 Generality—Not Just for DiscreteRandom Variables . . . . . . . . . . . . . . 38
3.4.1.1 What Is It? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.4.2 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.4.3 Existence of the Expected Value . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.4.4 Computation and Properties of Expected Value . . . . . . . . . . . . . . . . . 40
3.4.5 “Mailing Tubes” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.4.6 Casinos, Insurance Companies and “Sum Users,” Compared to Others . . . . 45
3.5 Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.5.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.5.2 Central Importance of the Concept of Variance . . . . . . . . . . . . . . . . . 50
3.5.3 Intuition Regarding the Size of Var(X) . . . . . . . . . . . . . . . . . . . . . . 50
CONTENTS iii
3.5.3.1 Chebychev’s Inequality . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.5.3.2 The Coefficient of Variation . . . . . . . . . . . . . . . . . . . . . . . 50
3.6 A Useful Fact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.7 Covariance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.8 Indicator Random Variables, and Their Means and Variances . . . . . . . . . . . . . 52
3.8.1 Example: Return Time for Library Books . . . . . . . . . . . . . . . . . . . . 53
3.8.2 Example: Indicator Variables in a Committee Problem . . . . . . . . . . . . . 54
3.9 Expected Value, Etc. in the ALOHA Example . . . . . . . . . . . . . . . . . . . . . 55
3.10 Example: Measurements at Different Ages . . . . . . . . . . . . . . . . . . . . . . . . 56
3.11 Example: Bus Ridership Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.12 Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.12.1 Example: Toss Coin Until First Head . . . . . . . . . . . . . . . . . . . . . . 58
3.12.2 Example: Sum of Two Dice . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
3.12.3 Example: Watts-Strogatz Random Graph Model . . . . . . . . . . . . . . . . 59
3.12.3.1 The Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
3.12.3.2 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.13 Parameteric Families of pmfs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.13.1 Parameteric Families of Functions . . . . . . . . . . . . . . . . . . . . . . . . 60
3.13.2 The Case of Importance to Us: Parameteric Families of pmfs . . . . . . . . . 61
3.13.3 The Geometric Family of Distributions . . . . . . . . . . . . . . . . . . . . . . 62
3.13.3.1 R Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.13.3.2 Example: a Parking Space Problem . . . . . . . . . . . . . . . . . . 65
3.13.4 The Binomial Family of Distributions . . . . . . . . . . . . . . . . . . . . . . 67
3.13.4.1 R Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.13.4.2 Example: Flipping Coins with Bonuses . . . . . . . . . . . . . . . . 69
3.13.4.3 Example: Analysis of Social Networks . . . . . . . . . . . . . . . . . 70
3.13.5 The Negative Binomial Family of Distributions . . . . . . . . . . . . . . . . . 71
iv CONTENTS
3.13.5.1 R Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.13.5.2 Example: Backup Batteries . . . . . . . . . . . . . . . . . . . . . . . 72
3.13.6 The Poisson Family of Distributions . . . . . . . . . . . . . . . . . . . . . . . 73
3.13.6.1 R Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.13.7 The Power Law Family of Distributions . . . . . . . . . . . . . . . . . . . . . 74
3.13.7.1 The Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.13.7.2 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.14 Recognizing Some Parametric Distributions When You See Them . . . . . . . . . . . 75
3.14.1 Example: a Coin Game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
3.14.2 Example: Tossing a Set of Four Coins . . . . . . . . . . . . . . . . . . . . . . 77
3.14.3 Example: the ALOHA Example Again . . . . . . . . . . . . . . . . . . . . . . 78
3.15 Example: the Bus Ridership Problem Again . . . . . . . . . . . . . . . . . . . . . . . 79
3.16 Multivariate Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
3.17 Iterated Expectations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.17.1 The Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.17.2 Example: Coin and Die Game . . . . . . . . . . . . . . . . . . . . . . . . . . 82
3.18 A Cautionary Tale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3.18.1 Trick Coins, Tricky Example . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3.18.2 Intuition in Retrospect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
3.18.3 Implications for Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
3.19 Why Not Just Do All Analysis by Simulation? . . . . . . . . . . . . . . . . . . . . . 84
3.20 Proof of Chebychev’s Inequality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
3.21 Reconciliation of Math and Intuition (optional section) . . . . . . . . . . . . . . . . . 86
4 Introduction to Discrete Markov Chains 93
4.1 Matrix Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
4.2 Example: Die Game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
CONTENTS v
4.3 Long-Run State Probabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
4.3.1 Calculation of π . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
4.4 Example: 3-Heads-in-a-Row Game . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
4.5 Example: ALOHA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
4.6 Example: Bus Ridership Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
4.7 Example: an Inventory Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
5 Continuous Probability Models 103
5.1 A Random Dart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
5.2 Continuous Random Variables Are “Useful Unicorns” . . . . . . . . . . . . . . . . . 104
5.3 But Now We Have a Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
5.4 Density Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
5.4.1 Motivation, Definition and Interpretation . . . . . . . . . . . . . . . . . . . . 108
5.4.2 Properties of Densities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.4.3 A First Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
5.5 Iterated Expectations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
5.5.1 The Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
5.5.2 Example: Another Coin Game . . . . . . . . . . . . . . . . . . . . . . . . . . 114
5.6 Famous Parametric Families of Continuous Distributions . . . . . . . . . . . . . . . . 114
5.6.1 The Uniform Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
5.6.1.1 Density and Properties . . . . . . . . . . . . . . . . . . . . . . . . . 114
5.6.1.2 R Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
5.6.1.3 Example: Modeling of Disk Performance . . . . . . . . . . . . . . . 115
5.6.1.4 Example: Modeling of Denial-of-Service Attack . . . . . . . . . . . . 116
5.6.2 The Normal (Gaussian) Family of Continuous Distributions . . . . . . . . . . 116
5.6.2.1 Density and Properties . . . . . . . . . . . . . . . . . . . . . . . . . 116
5.6.3 The Chi-Squared Family of Distributions . . . . . . . . . . . . . . . . . . . . 117
vi CONTENTS
5.6.3.1 Density and Properties . . . . . . . . . . . . . . . . . . . . . . . . . 117
5.6.3.2 Example: Error in Pin Placement . . . . . . . . . . . . . . . . . . . 117
5.6.3.3 Importance in Modeling . . . . . . . . . . . . . . . . . . . . . . . . . 118
5.6.4 The Exponential Family of Distributions . . . . . . . . . . . . . . . . . . . . . 118
5.6.4.1 Density and Properties . . . . . . . . . . . . . . . . . . . . . . . . . 119
5.6.4.2 R Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
5.6.4.3 Example: Refunds on Failed Components . . . . . . . . . . . . . . . 119
5.6.4.4 Example: Garage Parking Fees . . . . . . . . . . . . . . . . . . . . . 120
5.6.4.5 Importance in Modeling . . . . . . . . . . . . . . . . . . . . . . . . . 120
5.6.5 The Gamma Family of Distributions . . . . . . . . . . . . . . . . . . . . . . . 121
5.6.5.1 Density and Properties . . . . . . . . . . . . . . . . . . . . . . . . . 121
5.6.5.2 Example: Network Buffer . . . . . . . . . . . . . . . . . . . . . . . . 122
5.6.5.3 Importance in Modeling . . . . . . . . . . . . . . . . . . . . . . . . . 122
5.6.6 The Beta Family of Distributions . . . . . . . . . . . . . . . . . . . . . . . . . 124
5.6.6.1 Density Etc. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
5.6.6.2 Importance in Modeling . . . . . . . . . . . . . . . . . . . . . . . . . 125
5.7 Choosing a Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
5.8 A General Method for Simulating a Random Variable . . . . . . . . . . . . . . . . . 126
5.9 Example: Writing a Set of R Functions for a Certain Power Family . . . . . . . . . . 126
5.10 Multivariate Densities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
5.11 “Hybrid” Continuous/Discrete Distributions . . . . . . . . . . . . . . . . . . . . . . . 128
6 The Normal Family of Distributions 131
6.1 Density and Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
6.1.1 Closure Under Affine Transformation . . . . . . . . . . . . . . . . . . . . . . . 131
6.1.2 Closure Under Independent Summation . . . . . . . . . . . . . . . . . . . . . 132
6.1.3 Evaluating Normal cdfs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
CONTENTS vii
6.2 Example: Network Intrusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
6.3 Example: Class Enrollment Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
6.4 More on the Jill Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
6.5 Example: River Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
6.6 Example: Upper Tail of a Light Bulb Distribution . . . . . . . . . . . . . . . . . . . 137
6.7 The Central Limit Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
6.8 Example: Cumulative Roundoff Error . . . . . . . . . . . . . . . . . . . . . . . . . . 138
6.9 Example: R Evaluation of a Central Limit Theorem Approximation . . . . . . . . . 138
6.10 Example: Bug Counts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
6.11 Example: Coin Tosses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
6.12 Museum Demonstration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
6.13 Optional topic: Formal Statement of the CLT . . . . . . . . . . . . . . . . . . . . . . 141
6.14 Importance in Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
6.15 The Multivariate Normal Family . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
7 The Exponential Distributions 145
7.1 Connection to the Poisson Distribution Family . . . . . . . . . . . . . . . . . . . . . 145
7.2 Memoryless Property of Exponential Distributions . . . . . . . . . . . . . . . . . . . 147
7.2.1 Derivation and Intuition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
7.2.2 Uniquely Memoryless . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
7.2.3 Example: “Nonmemoryless” Light Bulbs. . . . . . . . . . . . . . . . . . . . . 149
7.3 Example: Minima of Independent Exponentially Distributed Random Variables . . . 149
7.3.1 Example: Computer Worm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
7.3.2 Example: Electronic Components . . . . . . . . . . . . . . . . . . . . . . . . . 153
8 Introduction to Continuous-Time Markov Chains 155
8.1 Continuous-Time Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
viii CONTENTS
8.2 Holding-Time Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
8.2.1 The Notion of “Rates” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
8.3 Stationary Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
8.3.1 Intuitive Derivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
8.3.2 Computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
8.4 Example: Machine Repair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
8.5 Example: Migration in a Social Network . . . . . . . . . . . . . . . . . . . . . . . . . 159
9 Mixture Models 161
9.1 The Old Trick Coin Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
9.2 Generating Random Variates from a Mixture Distribution . . . . . . . . . . . . . . . 163
9.2.1 The Law of Total Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . 164
9.2.1.1 Conditional Expected Value As a Random Variable . . . . . . . . . 164
9.2.1.2 Famous Formula: Theorem of Total Expectation . . . . . . . . . . . 165
9.3 What About the Variance? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
9.4 Example: Trapped Miner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
9.5 Example: More on Flipping Coins with Bonuses . . . . . . . . . . . . . . . . . . . . 167
9.6 Example: Analysis of Hash Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
9.7 The EM Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
9.8 Mean and Variance of Random Variables Having Mixture Distributions . . . . . . . 170
9.9 Example: Two Kinds of Batteries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
9.10 Example: Overdispersion Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
9.11 Vector Space Interpretations (for the mathematically adventurous only) . . . . . . . 173
9.11.1 Properties of Correlation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
9.11.2 Conditional Expectation As a Projection . . . . . . . . . . . . . . . . . . . . 174
9.12 Proof of the Law of Total Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . 177