Probability and random processes

735 Pages·2006·9.394 MB·English

by  Venkatarama Krishnan

B
PROBABILITY AND
RANDOM PROCESSES
WILEY SURVIVAL GUIDES IN ENGINEERING
AND SCIENCE
Emmanuel Desurvire, Editor
Wiley Survival Guide in Global Telecommunications: Signaling Principles,
Network Protocols, and Wireless Systems EmmanuelDesurvire
Wiley Survival Guide in Global Telecommunications: Broadband Access,
OpticalComponents and Networks, and Cryptography
EmmanuelDesurvire
Fiber to the Home: The New Empowerment PaulE.Green,Jr.
ProbabilityandRandom Processes VenkataramaKrishnan
B
PROBABILITY AND
RANDOM PROCESSES
Venkatarama Krishnan
Professor Emeritusof Electrical Engineering
UniversityofMassachusetts Lowell
AJohn Wiley & Sons, Inc.,Publication
Copyright#2006byJohnWiley&Sons,Inc.Allrightsreserved
PublishedbyJohnWiley&Sons,Inc.,Hoboken,NewJersey
PublishedsimultaneouslyinCanada
Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmittedinanyform
orbyanymeans,electronic,mechanical,photocopying,recording,scanning,orotherwise,exceptas
permittedunderSection107or108ofthe1976UnitedStatesCopyrightAct,withouteithertheprior
writtenpermissionofthePublisher,orauthorizationthroughpaymentoftheappropriateper-copy
feetotheCopyrightClearanceCenter,Inc.,222RosewoodDrive,Danvers,MA01923,[phone],
fax[phone],oronthewebatwww.copyright.com.RequeststothePublisherforpermission
should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken,
NJ07030,[phone],fax[phone],oronlineathttp://www.wiley.com/go/permission.
LimitofLiability/DisclaimerofWarranty:Whilethepublisherandauthorhaveusedtheirbestefforts
inpreparingthisbook,theymakenorepresentationsorwarrantieswithrespecttotheaccuracyor
completenessofthecontentsofthisbookandspecificallydisclaimanyimpliedwarrantiesof
merchantabilityorfitnessforaparticularpurpose.Nowarrantymaybecreatedorextendedbysales
representativesorwrittensalesmaterials.Theadviceandstrategiescontainedhereinmaynotbe
suitableforyoursituation.Youshouldconsultwithaprofessionalwhereappropriate.Neitherthe
publishernorauthorshallbeliableforanylossofprofitoranyothercommercialdamages,including
butnotlimitedtospecial,incidental,consequential,orotherdamages.
Forgeneralinformationonourotherproductsandservicesorfortechnicalsupport,pleasecontactourCustomer
Care Department within the United States at [phone], outside the United States at [phone]
orfax(317)572-4002.
Wileyalsopublishesitsbooksinavarietyofelectronicformats.Somecontentthatappearsinprint
maynotbeavailableinelectronicformats.FormoreinformationaboutWileyproducts,visitourwebsiteat
www.wiley.com.
LibraryofCongressCataloging-in-PublicationData:
Krishnan,Venkatarama,
Probabilityandrandomprocesses/byVenkataramaKrishnan.
p.cm.—(Wileysurvivalguidesinengineeringandscience)
Includesbibliographicalreferencesandindex.
ISBN-13:978-0-471-70354-9(acid-freepaper)
ISBN-10:0-471-70354-0(acid-freepaper)
1.Probabilities.2.Stochasticprocesses.3.Engineering—Statisticalmethods.
4.Science—Statisticalmethods.I.Title.II.Series
QA273.K742006
519.2–dc22 2005057726
PrintedintheUnitedStatesofAmerica
10 9 8 7 6 5 4 3 2 1

Contents
Preface, xi 4.6 Hypergeometric Distribution, 46
4.7 Poisson Distribution, 48
CHAPTER 1 4.8 Logarithmic Distribution, 55
Sets,Fields,andEvents, 1
4.9 SummaryofDiscreteDistributions, 62
1.1 Set Definitions, 1
1.2 Set Operations, 3 CHAPTER5
RandomVariables, 64
1.3 Set Algebras,Fields, andEvents, 8
5.1 DefinitionofRandom Variables, 64
CHAPTER 2
5.2 Determinationof Distribution and
ProbabilitySpaceandAxioms, 10
Density Functions, 66
2.1 Probability Space, 10
5.3 Properties ofDistribution andDensity
2.2 Conditional Probability, 14 Functions, 73
2.3 Independence, 18 5.4 DistributionFunctions from Density
Functions, 75
2.4 Total Probability and Bayes’
Theorem, 20
CHAPTER6
CHAPTER 3 ContinuousRandomVariablesandBasic
BasicCombinatorics, 25 Distributions, 79
3.1 Basic Counting Principles, 25 6.1 Introduction, 79
3.2 Permutations, 26 6.2 Uniform Distribution, 79
3.3 Combinations, 28 6.3 ExponentialDistribution, 80
6.4 NormalorGaussian Distribution, 84
CHAPTER 4
DiscreteDistributions, 37
CHAPTER7
4.1 BernoulliTrials, 37 OtherContinuousDistributions, 95
4.2 Binomial Distribution, 38 7.1 Introduction, 95
4.3 MultinomialDistribution, 41 7.2 Triangular Distribution, 95
4.4 Geometric Distribution, 42 7.3 Laplace Distribution, 96
4.5 Negative Binomial Distribution, 44 7.4 Erlang Distribution, 97
vii
viii Contents
7.5 GammaDistribution, 99 10.4 Higher-Order Moments, 153
7.6 WeibullDistribution, 101 10.5 Bivariate Gaussian, 154
7.7 Chi-Square Distribution, 102
CHAPTER 11
7.8 Chi andOther Allied Distributions, CharacteristicFunctionsandGenerating
104 Functions, 155
7.9 Student-tDensity, 110 11.1 Characteristic Functions, 155
7.10 Snedecor F Distribution, 111 11.2 Examplesof Characteristic
Functions, 157
7.11 Lognormal Distribution, 112
11.3 GeneratingFunctions, 161
7.12 Beta Distribution, 114
11.4 Examplesof Generating
7.13 CauchyDistribution, 115
Functions, 162
7.14 Pareto Distribution, 117
11.5 Moment Generating Functions, 164
7.15 Gibbs Distribution, 118
11.6 Cumulant Generating Functions, 167
7.16 MixedDistributions, 118
11.7 Table ofMeansand Variances, 170
7.17 Summary ofDistributions of
Continuous Random Variables, 119 CHAPTER 12
FunctionsofaSingleRandomVariable, 173
CHAPTER 8 12.1 Random Variable g(X), 173
ConditionalDensitiesandDistributions, 122
12.2 Distribution of Y¼g(X), 174
8.1 Conditional Distribution andDensity
for P(A)=0, 122 12.3 Direct Determination ofDensity
f (y)from f (x), 194
Y X
8.2 Conditional Distribution andDensity
for P(A)¼0, 126 12.4 Inverse Problem: Finding g(x) Given
f (x)and f (y), 200
X Y
8.3 Total Probability andBayes’ Theorem
for Densities, 131 12.5 Momentsof aFunctionof
aRandomVariable, 202
CHAPTER 9
JointDensitiesandDistributions, 135 CHAPTER 13
FunctionsofMultipleRandom
9.1 Joint DiscreteDistribution
Variables, 206
Functions, 135
13.1 Function ofTwo Random Variables,
9.2 Joint Continuous Distribution Z¼g(X,Y), 206
Functions, 136
13.2 Two Functions ofTwo Random
9.3 Bivariate GaussianDistributions, 144 Variables,Z¼g(X,Y),
W¼h(X,Y), 222
CHAPTER 10
13.3 Direct Determination ofJointDensity
MomentsandConditionalMoments, 146
f (z,w) fromf (x,y), 227
ZW XY
10.1 Expectations, 146
13.4 SolvingZ¼g(X,Y)UsinganAuxiliary
10.2 Variance, 149 Random Variable, 233
10.3 Means andVariances ofSome 13.5 MultipleFunctionsof
Distributions, 150 RandomVariables, 238
Contents ix
CHAPTER 14 17.4 Diagonalization of
Inequalities,Convergences,andLimit Covariance Matrices, 330
Theorems, 241
17.5 Simultaneous Diagonalizationof
14.1 Degenerate Random Variables, 241 Covariance Matrices, 334
14.2 Chebyshev andAllied 17.6 LinearEstimation of Vector
Inequalities, 242 Variables, 337
14.3 Markov Inequality, 246
CHAPTER18
14.4 Chernoff Bound, 248
EstimationTheory, 340
14.5 Cauchy–SchwartzInequality, 251
18.1 Criteria ofEstimators, 340
14.6 Jensen’s Inequality, 254
18.2 Estimation ofRandom
14.7 Convergence Concepts, 256 Variables, 342
14.8 Limit Theorems, 259 18.3 Estimation ofParameters (Point
Estimation), 350
CHAPTER 15
ComputerMethodsforGenerating 18.4 IntervalEstimation (Confidence
RandomVariates, 264 Intervals), 364
15.1 Uniform-DistributionRandom 18.5 Hypothesis Testing(Binary), 373
Variates, 264
18.6 BayesianEstimation, 384
15.2 Histograms, 266
15.3 Inverse Transformation CHAPTER19
Techniques, 269 RandomProcesses, 406
15.4 Convolution Techniques, 279 19.1 Basic Definitions, 406
15.5 Acceptance–Rejection 19.2 Stationary Random Processes, 420
Techniques, 280
19.3 ErgodicProcesses, 439
CHAPTER 16 19.4 Estimation ofParameters of
ElementsofMatrixAlgebra, 284 Random Processes, 445
16.1 Basic Theory ofMatrices, 284 19.5 PowerSpectralDensity, 472
16.2 Eigenvalues and
Eigenvectors of Matrices, 293 CHAPTER20
ClassificationofRandomProcesses, 490
16.3 Vectors and Matrix
Differentiations, 301 20.1 Specifications of Random
Processes, 490
16.4 Block Matrices, 308
20.2 PoissonProcess, 492
CHAPTER 17
20.3 BinomialProcess, 505
RandomVectorsand
Mean-SquareEstimation, 311 20.4 Independent Increment Process, 507
17.1 Distributions and Densities, 311 20.5 Random-WalkProcess, 512
17.2 Momentsof Random Vectors, 319 20.6 GaussianProcess, 521
17.3 Vector GaussianRandom 20.7 Wiener Process (Brownian
Variables, 323 Motion), 523
x Contents
20.8 Markov Process, 527 CHAPTER 23
ProbabilisticMethodsinTransmission
20.9 Markov Chain, 536
Tomography, 666
20.10 MartingaleProcess, 551 23.1 Introduction, 666
20.11 Periodic Random Process, 557 23.2 Stochastic Model, 667
20.12 Aperiodic Random Process 23.3 Stochastic Estimation Algorithm, 671
(Karhunen–Loeve Expansion), 563
23.4 PriorDistribution PfMg, 674
23.5 Computer Simulation, 676
CHAPTER 21
RandomProcessesandLinear 23.6 Results and Conclusions, 678
Systems, 574
23.7 Discussionof Results, 681
21.1 Review of Linear Systems, 574
23.8 Referencesfor Chapter 23, 681
21.2 Random Processes through
LinearSystems, 578
APPENDIXES
21.3 LinearFilters, 592
A AFourier TransformTables, 683
21.4 Bandpass Stationary Random
B Cumulative Gaussian Tables, 687
Processes, 606
C InverseCumulative Gaussian Tables, 692
D Inverse Chi-Square Tables, 694
CHAPTER 22
WeinerandKalmanFilters, 625 E Inverse Student-tTables, 701
22.1 Review of Orthogonality F Cumulative Poisson Distribution, 704
Principle, 625
G Cumulative BinomialDistribution, 708
22.2 Wiener Filtering, 627
References, 714
22.3 Discrete Kalman Filter, 647
22.4 Continuous Kalman Filter, 660 Index, 716
Preface
Many good textbooks exist on probability and random processes written at the under-
graduate level to the research level. However, there is no one handy and ready book
that explains most of the essential topics, such as random variables and most of their
frequently used discrete and continuous probability distribution functions; moments,
transformation, and convergences of random variables; characteristic and generating
functions; estimation theory and the associated orthogonality principle; vector random
variables;randomprocessesandtheirautocovarianceandcross-covariancefunctions;sta-
tionarity concepts; and random processes through linear systems and the associated
Wiener and Kalman filters. Engineering practitioners and students alike have to delve
through several books to get the required formulas or tables either to complete a project
or to finish a homework assignment. This book may alleviate this difficulty to some
extent and provide access to a compendium of most distribution functions used by
communication engineers, queuing theory specialists, signal processing engineers, bio-
medical engineers, and physicists. Probability tables with accuracy up to nine decimal
places are given in the appendixes to enhance the utility of this book. A particular
feature is the presentation of commonly occurring Fourier transforms where both the
time and frequency functions are drawn toscale.
Mostofthetheoryhasbeenexplainedwithfiguresdrawntoscale.Tounderstandthe
theorybetter,morethan300examplesaregivenwitheverystepexplainedclearly.Follow-
ingtheadagethatafigureisworthmorethanathousandwords,mostoftheexamplesare
alsoillustratedwithfiguresdrawntoscale,resultinginmorethan400diagrams.Thisbook
willbeofparticularvaluetograduateandundergraduatestudentsinelectrical,computer,
and civil engineeringaswell as studentsinphysicsand applied mathematics for solving
homework assignments and projects. It will certainly be useful to communication and
signal processing engineers, and computer scientists in an industrial setting. It will also
serve as a good reference for research workers in biostatistics and financial market
analysis.
The salientfeaturesof thisbook are
. Functional andstatistical independenceof random variables are explained.
. Readyreferencetocommonlyoccurringdensityanddistributionfunctionsandtheir
meansand variancesis provided.
. A section on Benford’s logarithmic law, which is used in detecting tax fraud, is
included.
. More than 300 examples, many of them solved in different ways to illustrate the
theoryand various applications ofprobability, are presented.
. Most examples have been substantiated with graphsdrawnto scale.
xi