Table Of ContentPROBABILITY, STATISTICS,
AND RANDOM SIGNALS
PROBABILITY, STATISTICS,
AND RANDOM SIGNALS
CHARLES G. BONCELET JR.
UniversityofDelaware
NewYork•Oxford
OXFORDUNIVERSITYPRESS
OxfordUniversityPressisadepartmentoftheUniversityofOxford.
ItfurtherstheUniversity’sobjectiveofexcellenceinresearch,
scholarship,andeducationbypublishingworldwide.
Oxford NewYork
Auckland CapeTown DaresSalaam HongKong Karachi
KualaLumpur Madrid Melbourne MexicoCity Nairobi
NewDelhi Shanghai Taipei Toronto
Withofficesin
Argentina Austria Brazil Chile CzechRepublic France Greece
Guatemala Hungary Italy Japan Poland Portugal Singapore
SouthKorea Switzerland Thailand Turkey Ukraine Vietnam
Copyright©2016byOxfordUniversityPress
FortitlescoveredbySection112oftheU.S.HigherEducationOpportunity
Act,pleasevisitwww.oup.com/us/heforthelatestinformationabout
pricingandalternateformats.
PublishedbyOxfordUniversityPress.
198MadisonAvenue,NewYork,NY10016
http://www.oup.com
OxfordisaregisteredtrademarkofOxfordUniversityPress.
Allrightsreserved.Nopartofthispublicationmaybereproduced,
storedinaretrievalsystem,ortransmitted,inanyformorbyanymeans,
electronic,mechanical,photocopying,recording,orotherwise,
withoutthepriorpermissionofOxfordUniversityPress.
LibraryofCongressCataloginginPublicationData
Names:Boncelet,CharlesG.
Title:Probability,statistics,andrandomsignals/CharlesG.BonceletJr.
Description:NewYork:OxfordUniversityPress,[2017]|Series:TheOxfordseriesinelectricalandcomputer
engineering|Includesindex.
Identifiers:LCCN2015034908|ISBN9780190200510
Subjects:LCSH:Mathematicalstatistics–Textbooks.|Probabilities–Textbooks.|Electrical
engineering–Mathematics–Textbooks.
Classification:LCCQA276.18.B662017|DDC519.5–dc23LCrecordavailableat
http://lccn.loc.gov/2015034908
Printingnumber:9 8 7 6 5 4 3 2 1
PrintedintheUnitedStatesofAmerica
onacid-freepaper
CONTENTS
PREFACE xi
1 PROBABILITYBASICS 1
1.1 WhatIsProbability? 1
1.2 Experiments,Outcomes,andEvents 3
1.3 VennDiagrams 4
1.4 RandomVariables 5
1.5 BasicProbabilityRules 6
1.6 ProbabilityFormalized 9
1.7 SimpleTheorems 11
1.8 CompoundExperiments 15
1.9 Independence 16
1.10 Example:CanSCommunicateWithD? 17
1.10.1 ListAllOutcomes 18
1.10.2 ProbabilityofaUnion 19
1.10.3 ProbabilityoftheComplement 20
1.11 Example:NowCanSCommunicateWithD? 21
1.11.1 ABigTable 21
1.11.2 BreakIntoPieces 22
1.11.3 ProbabilityoftheComplement 23
1.12 ComputationalProcedures 23
Summary 24
Problems 25
2 CONDITIONALPROBABILITY 29
2.1 DefinitionsofConditionalProbability 29
2.2 LawofTotalProbabilityandBayesTheorem 32
2.3 Example:UrnModels 34
2.4 Example:ABinaryChannel 36
2.5 Example:DrugTesting 38
2.6 Example:ADiamondNetwork 40
Summary 41
Problems 42
3 ALITTLECOMBINATORICS 47
3.1 BasicsofCounting 47
3.2 NotesonComputation 52
v
vi CONTENTS
3.3 CombinationsandtheBinomialCoefficients 53
3.4 TheBinomialTheorem 54
3.5 MultinomialCoefficientandTheorem 55
3.6 TheBirthdayParadoxandMessageAuthentication 57
3.7 HypergeometricProbabilitiesandCardGames 61
Summary 66
Problems 67
4 DISCRETEPROBABILITIESANDRANDOMVARIABLES 75
4.1 ProbabilityMassFunctions 75
4.2 CumulativeDistributionFunctions 77
4.3 ExpectedValues 78
4.4 MomentGeneratingFunctions 83
4.5 SeveralImportantDiscretePMFs 85
4.5.1 UniformPMF 86
4.5.2 GeometricPMF 87
4.5.3 ThePoissonDistribution 90
4.6 GamblingandFinancialDecisionMaking 92
Summary 95
Problems 96
5 MULTIPLEDISCRETERANDOMVARIABLES 101
5.1 MultipleRandomVariablesandPMFs 101
5.2 Independence 104
5.3 MomentsandExpectedValues 105
5.3.1 ExpectedValuesforTwoRandomVariables 105
5.3.2 MomentsforTwoRandomVariables 106
5.4 Example:TwoDiscreteRandomVariables 108
5.4.1 MarginalPMFsandExpectedValues 109
5.4.2 Independence 109
5.4.3 JointCDF 110
5.4.4 TransformationsWithOneOutput 110
5.4.5 TransformationsWithSeveralOutputs 112
5.4.6 Discussion 113
5.5 SumsofIndependentRandomVariables 113
5.6 SampleProbabilities,Mean,andVariance 117
5.7 Histograms 119
5.8 EntropyandDataCompression 120
5.8.1 EntropyandInformationTheory 121
5.8.2 VariableLengthCoding 123
5.8.3 EncodingBinarySequences 127
5.8.4 MaximumEntropy 128
Summary 131
Problems 132
CONTENTS vii
6 BINOMIALPROBABILITIES 137
6.1 BasicsoftheBinomialDistribution 137
6.2 ComputingBinomialProbabilities 141
6.3 MomentsoftheBinomialDistribution 142
6.4 SumsofIndependentBinomialRandomVariables 144
6.5 DistributionsRelatedtotheBinomial 146
6.5.1 ConnectionsBetweenBinomialandHypergeometric
Probabilities 146
6.5.2 MultinomialProbabilities 147
6.5.3 TheNegativeBinomialDistribution 148
6.5.4 ThePoissonDistribution 149
6.6 BinomialandMultinomialEstimation 151
6.7 Alohanet 152
6.8 ErrorControlCodes 154
6.8.1 Repetition-by-ThreeCode 155
6.8.2 GeneralLinearBlockCodes 157
6.8.3 Conclusions 160
Summary 160
Problems 162
7 ACONTINUOUSRANDOMVARIABLE 167
7.1 BasicProperties 167
7.2 ExampleCalculationsforOneRandomVariable 171
7.3 SelectedContinuousDistributions 174
7.3.1 TheUniformDistribution 174
7.3.2 TheExponentialDistribution 176
7.4 ConditionalProbabilities 179
7.5 DiscretePMFsandDeltaFunctions 182
7.6 Quantization 184
7.7 AFinalWord 187
Summary 187
Problems 189
8 MULTIPLECONTINUOUSRANDOMVARIABLES 192
8.1 JointDensitiesandDistributionFunctions 192
8.2 ExpectedValuesandMoments 194
8.3 Independence 194
8.4 ConditionalProbabilitiesforMultipleRandomVariables 195
8.5 ExtendedExample:TwoContinuousRandomVariables 198
8.6 SumsofIndependentRandomVariables 202
8.7 RandomSums 205
8.8 GeneralTransformationsandtheJacobian 207
8.9 ParameterEstimationfortheExponentialDistribution 214
8.10 ComparisonofDiscreteandContinuousDistributions 214
viii CONTENTS
Summary 215
Problems 216
9 THEGAUSSIANANDRELATEDDISTRIBUTIONS 221
9.1 TheGaussianDistributionandDensity 221
9.2 QuantileFunction 227
9.3 MomentsoftheGaussianDistribution 228
9.4 TheCentralLimitTheorem 230
9.5 RelatedDistributions 235
9.5.1 TheLaplaceDistribution 236
9.5.2 TheRayleighDistribution 236
9.5.3 TheChi-SquaredandFDistributions 238
9.6 MultipleGaussianRandomVariables 240
9.6.1 IndependentGaussianRandomVariables 240
9.6.2 TransformationtoPolarCoordinates 241
9.6.3 TwoCorrelatedGaussianRandomVariables 243
9.7 Example:DigitalCommunicationsUsingQAM 246
9.7.1 Background 246
9.7.2 DiscreteTimeModel 247
9.7.3 MonteCarloExercise 253
9.7.4 QAMRecap 258
Summary 259
Problems 260
10 ELEMENTSOFSTATISTICS 265
10.1 ASimpleElectionPoll 265
10.2 EstimatingtheMeanandVariance 269
10.3 RecursiveCalculationoftheSampleMean 271
10.4 ExponentialWeighting 273
10.5 OrderStatisticsandRobustEstimates 274
10.6 EstimatingtheDistributionFunction 276
10.7 PMFandDensityEstimates 278
10.8 ConfidenceIntervals 280
10.9 SignificanceTestsandp-Values 282
10.10 IntroductiontoEstimationTheory 285
10.11 MinimumMeanSquaredErrorEstimation 289
10.12 BayesianEstimation 291
Problems 295
11 GAUSSIANRANDOMVECTORSANDLINEARREGRESSION 298
11.1 GaussianRandomVectors 298
11.2 LinearOperationsonGaussianRandomVectors 303
11.3 LinearRegression 304
11.3.1 LinearRegressioninDetail 305
CONTENTS ix
11.3.2 StatisticsoftheLinearRegressionEstimates 309
11.3.3 ComputationalIssues 311
11.3.4 LinearRegressionExamples 313
11.3.5 ExtensionsofLinearRegression 317
Summary 319
Problems 320
12 HYPOTHESISTESTING 324
12.1 HypothesisTesting:BasicPrinciples 324
12.2 Example:RadarDetection 326
12.3 HypothesisTestsandLikelihoodRatios 331
12.4 MAPTests 335
Summary 336
Problems 337
13 RANDOMSIGNALSANDNOISE 340
13.1 IntroductiontoRandomSignals 340
13.2 ASimpleRandomProcess 341
13.3 FourierTransforms 342
13.4 WSSRandomProcesses 346
13.5 WSSSignalsandLinearFilters 350
13.6 Noise 352
13.6.1 ProbabilisticPropertiesofNoise 352
13.6.2 SpectralPropertiesofNoise 353
13.7 Example:AmplitudeModulation 354
13.8 Example:DiscreteTimeWienerFilter 357
13.9 TheSamplingTheoremforWSSRandomProcesses 357
13.9.1 Discussion 358
13.9.2 Example:Figure13.4 359
13.9.3 ProofoftheRandomSamplingTheorem 361
Summary 362
Problems 364
14 SELECTEDRANDOMPROCESSES 366
14.1 TheLightbulbProcess 366
14.2 ThePoissonProcess 368
14.3 MarkovChains 372
14.4 KalmanFilter 381
14.4.1 TheOptimalFilterandExample 381
14.4.2 QRMethodAppliedtotheKalmanFilter 384
Summary 386
Problems 388
