Fundamental Statistical Inference A Computational Approach

572 Pages·2018·12.89 MB·English

by  Marc S. Paolella

Fundamental Statistical
Inference
WILEYSERIESINPROBABILITYANDSTATISTICS
EstablishedbyWalterA.ShewhartandSamuelS.Wilks
Editors:DavidJ.Balding,NoelA.C.Cressie,GarrettM.Fitzmaurice,
GeofH.Givens,HarveyGoldstein,GeertMolenberghs,DavidW.Scott,
AdrianF.M.Smith,RueyS.Tsay
EditorsEmeriti:J.StuartHunter,IainM.Johnstone,JosephB.Kadane,
JozefL.Teugels
TheWileySeriesinProbabilityandStatisticsiswellestablishedandauthoritative.Itcovers
manytopicsofcurrentresearchinterestinbothpureandappliedstatisticsandprobability
theory.Writtenbyleadingstatisticiansandinstitutions,thetitlesspanbothstate-of-the-art
developmentsinthefieldandclassicalmethods.
Reflectingthewiderangeofcurrentresearchinstatistics,theseriesencompassesapplied,
methodological and theoretical statistics, ranging from applications and new techniques
made possible by advances in computerized practice to rigorous treatment of theoreti-
cal approaches. This series provides essential and invaluable reading for all statisticians,
whetherinacademia,industry,government,orresearch.
Acompletelistoftitlesinthisseriescanbefoundathttp://www.wiley.com/go/wsps
Fundamental Statistical
Inference
A Computational Approach
MarcS.Paolella
DepartmentofBankingandFinance
UniversityofZurich
Switzerland
(cid:2)
Thiseditionfirstpublished2018
©2018JohnWiley&SonsLtd
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmitted,in
anyformorbyanymeans,electronic,mechanical,photocopying,recordingorotherwise,exceptaspermittedby
law.Adviceonhowtoobtainpermissiontoreusematerialfromthistitleisavailableathttp://www.wiley.com/
go/permissions.
TherightofMarcS.Paolellatobeidentifiedastheauthorofthisworkhasbeenassertedinaccordancewithlaw.
RegisteredOffices
JohnWiley&Sons,Inc.,111RiverStreet,Hoboken,NJ07030,USA
JohnWiley&SonsLtd,TheAtrium,SouthernGate,Chichester,WestSussex,PO198SQ,UK
EditorialOffice
9600GarsingtonRoad,Oxford,OX42DQ,UK
Fordetailsofourglobaleditorialoffices,customerservices,andmoreinformationaboutWileyproductsvisitus
atwww.wiley.com.
Wileyalsopublishesitsbooksinavarietyofelectronicformatsandbyprint-on-demand.Somecontentthat
appearsinstandardprintversionsofthisbookmaynotbeavailableinotherformats.
LimitofLiability/DisclaimerofWarranty
Whilethepublisherandauthorshaveusedtheirbesteffortsinpreparingthiswork,theymakenorepresentations
orwarrantieswithrespecttotheaccuracyorcompletenessofthecontentsofthisworkandspecificallydisclaim
allwarranties,includingwithoutlimitationanyimpliedwarrantiesofmerchantabilityorfitnessforaparticular
purpose.Nowarrantymaybecreatedorextendedbysalesrepresentatives,writtensalesmaterialsor
promotionalstatementsforthiswork.Thefactthatanorganization,website,orproductisreferredtointhis
(cid:2) (cid:2)
workasacitationand/orpotentialsourceoffurtherinformationdoesnotmeanthatthepublisherandauthors
endorsetheinformationorservicestheorganization,website,orproductmayprovideorrecommendationsit
maymake.Thisworkissoldwiththeunderstandingthatthepublisherisnotengagedinrenderingprofessional
services.Theadviceandstrategiescontainedhereinmaynotbesuitableforyoursituation.Youshouldconsult
withaspecialistwhereappropriate.Further,readersshouldbeawarethatwebsiteslistedinthisworkmayhave
changedordisappearedbetweenwhenthisworkwaswrittenandwhenitisread.Neitherthepublishernor
authorsshallbeliableforanylossofprofitoranyothercommercialdamages,includingbutnotlimitedto
special,incidental,consequential,orotherdamages.
LibraryofCongressCataloging-in-PublicationDataappliedfor
HardbackISBN:9781119417866
CoverdesignbyWiley
Coverimages:CourtesyofMarcS.Paolella
Setin10/12ptTimesLTStdbySPiGlobal,Chennai,India
10 9 8 7 6 5 4 3 2 1
(cid:2)
Contents
PREFACE xi
PARTI ESSENTIALCONCEPTSINSTATISTICS
1 IntroducingPointandIntervalEstimation 3
1.1 PointEstimation / 4
1.1.1 BernoulliModel / 4
1.1.2 GeometricModel / 6
1.1.3 SomeRemarksonBiasandConsistency / 11
1.2 IntervalEstimationviaSimulation / 12
1.3 IntervalEstimationviatheBootstrap / 18
1.3.1 ComputationandComparisonwithParametricBootstrap / 18
1.3.2 ApplicationtoBernoulliModelandModification / 20
1.3.3 DoubleBootstrap / 24
1.3.4 DoubleBootstrapwithAnalyticInnerLoop / 26
1.4 BootstrapConfidenceIntervalsintheGeometricModel / 31
1.5 Problems / 35
2 GoodnessofFitandHypothesisTesting 37
2.1 EmpiricalCumulativeDistributionFunction / 38
2.1.1 TheGlivenko–CantelliTheorem / 38
2.1.2 ProofsoftheGlivenko–CantelliTheorem / 41
v
vi CONTENTS
2.1.3 ExamplewithContinuousDataandApproximateConfidence
Intervals / 45
2.1.4 ExamplewithDiscreteDataandApproximateConfidence
Intervals / 49
2.2 ComparingParametricandNonparametricMethods / 52
2.3 Kolmogorov–SmirnovDistanceandHypothesisTesting / 57
2.3.1 TheKolmogorov–SmirnovandAnderson–DarlingStatistics / 57
2.3.2 SignificanceandHypothesisTesting / 59
2.3.3 Small-SampleCorrection / 63
2.4 TestingNormalitywithKDandAD / 65
2.5 TestingNormalitywithW2andU2 / 68
2.6 TestingtheStableParetianDistributionalAssumption:FirstAttempt / 69
2.7 Two-SampleKolmogorovTest / 73
2.8 Moreon(Moron?)HypothesisTesting / 74
2.8.1 Explanation / 75
2.8.2 MisuseofHypothesisTesting / 77
2.8.3 UseandMisuseofp-Values / 79
2.9 Problems / 82
3 Likelihood 85
3.1 Introduction / 85
3.1.1 ScalarParameterCase / 87
3.1.2 VectorParameterCase / 92
3.1.3 RobustnessandtheMCDEstimator / 100
3.1.4 AsymptoticPropertiesoftheMaximumLikelihoodEstimator / 102
3.2 Cramér–RaoLowerBound / 107
3.2.1 UnivariateCase / 108
3.2.2 MultivariateCase / 111
3.3 ModelSelection / 114
3.3.1 ModelMisspecification / 114
3.3.2 TheLikelihoodRatioStatistic / 117
3.3.3 UseofInformationCriteria / 119
3.4 Problems / 120
4 NumericalOptimization 123
4.1 RootFinding / 123
4.1.1 OneParameter / 124
4.1.2 SeveralParameters / 131
4.2 ApproximatingtheDistributionoftheMaximumLikelihoodEstimator / 135
4.3 GeneralNumericalLikelihoodMaximization / 136
CONTENTS vii
4.3.1 Newton–RaphsonandQuasi-NewtonMethods / 137
4.3.2 ImposingParameterRestrictions / 140
4.4 EvolutionaryAlgorithms / 145
4.4.1 DifferentialEvolution / 146
4.4.2 CovarianceMatrixAdaptionEvolutionaryStrategy / 149
4.5 Problems / 155
5 MethodsofPointEstimation 157
5.1 UnivariateMixedNormalDistribution / 157
5.1.1 Introduction / 157
5.1.2 SimulationofUnivariateMixtures / 160
5.1.3 DirectLikelihoodMaximization / 161
5.1.4 UseoftheEMAlgorithm / 169
5.1.5 Shrinkage-TypeEstimation / 174
5.1.6 Quasi-BayesianEstimation / 176
5.1.7 ConfidenceIntervals / 178
5.2 AlternativePointEstimationMethodologies / 184
5.2.1 MethodofMomentsEstimator / 185
5.2.2 UseofGoodness-of-FitMeasures / 190
5.2.3 QuantileLeastSquares / 191
5.2.4 PearsonMinimumChi-Square / 193
5.2.5 EmpiricalMomentGeneratingFunctionEstimator / 195
5.2.6 EmpiricalCharacteristicFunctionEstimator / 198
5.3 ComparisonofMethods / 199
5.4 APrimeronShrinkageEstimation / 200
5.5 Problems / 202
PARTII FURTHERFUNDAMENTALCONCEPTSINSTATISTICS
6 Q-QPlotsandDistributionTesting 209
6.1 P-PPlotsandQ-QPlots / 209
6.2 NullBands / 211
6.2.1 DefinitionandMotivation / 211
6.2.2 PointwiseNullBandsviaSimulation / 212
6.2.3 AsymptoticApproximationofPointwiseNullBands / 213
6.2.4 MappingPointwiseandSimultaneousSignificanceLevels / 215
6.3 Q-QTest / 217
6.4 FurtherP-PandQ-QTypePlots / 219
6.4.1 (Horizontal)StabilizedP-PPlots / 219
viii CONTENTS
6.4.2 ModifiedS-PPlots / 220
6.4.3 MSPTestforNormality / 224
6.4.4 ModifiedPercentile(Fowlkes-MP)Plots / 228
6.5 FurtherTestsforCompositeNormality / 231
6.5.1 Motivation / 232
6.5.2 Jarque–BeraTest / 234
6.5.3 ThreePowerful(andMoreRecent)NormalityTests / 237
6.5.4 TestingGoodnessofFitviaBinning:Pearson’sX2 Test / 240
P
6.6 CombiningTestsandPowerEnvelopes / 247
6.6.1 CombiningTests / 248
6.6.2 PowerComparisonsforTestingCompositeNormality / 252
6.6.3 MostPowerfulTestsandPowerEnvelopes / 252
6.7 DetailsofaFailedAttempt / 255
6.8 Problems / 260
7 UnbiasedPointEstimationandBiasReduction 269
7.1 Sufficiency / 269
7.1.1 Introduction / 269
7.1.2 Factorization / 272
7.1.3 MinimalSufficiency / 276
7.1.4 TheRao–BlackwellTheorem / 283
7.2 CompletenessandtheUniformlyMinimumVarianceUnbiasedEstimator / 286
7.3 AnExamplewithi.i.d.GeometricData / 289
7.4 MethodsofBiasReduction / 293
7.4.1 TheBias-FunctionApproach / 293
7.4.2 Median-UnbiasedEstimation / 296
7.4.3 Mode-AdjustedEstimator / 297
7.4.4 TheJackknife / 302
7.5 Problems / 305
8 AnalyticIntervalEstimation 313
8.1 Definitions / 313
8.2 PivotalMethod / 315
8.2.1 ExactPivots / 315
8.2.2 AsymptoticPivots / 318
8.3 IntervalsAssociatedwithNormalSamples / 319
8.3.1 SingleSample / 319
8.3.2 PairedSample / 320
8.3.3 TwoIndependentSamples / 322
8.3.4 Welch’sMethodfor𝜇 −𝜇 when𝜎2 ≠𝜎2 / 323
1 2 1 2
8.3.5 Satterthwaite’sApproximation / 324
CONTENTS ix
8.4 CumulativeDistributionFunctionInversion / 326
8.4.1 ContinuousCase / 326
8.4.2 DiscreteCase / 330
8.5 ApplicationoftheNonparametricBootstrap / 334
8.6 Problems / 337
PARTIII ADDITIONALTOPICS
9 InferenceinaHeavy-TailedContext 341
9.1 EstimatingtheMaximallyExistingMoment / 342
9.2 APrimeronTailEstimation / 346
9.2.1 Introduction / 346
9.2.2 TheHillEstimator / 346
9.2.3 UsewithStableParetianData / 349
9.3 NoncentralStudent’stEstimation / 351
9.3.1 Introduction / 351
9.3.2 DirectDensityApproximation / 352
9.3.3 Quantile-BasedTableLookupEstimation / 353
9.3.4 ComparisonofNCTEstimators / 354
9.4 AsymmetricStableParetianEstimation / 358
9.4.1 Introduction / 358
9.4.2 TheHintEstimator / 359
9.4.3 MaximumLikelihoodEstimation / 360
9.4.4 TheMcCullochEstimator / 361
9.4.5 TheEmpiricalCharacteristicFunctionEstimator / 364
9.4.6 TestingforSymmetryintheStableModel / 366
9.5 TestingtheStableParetianDistribution / 368
9.5.1 TestBasedontheEmpiricalCharacteristicFunction / 368
9.5.2 SummabilityTestandModification / 371
9.5.3 ALHADI:The𝛼-HatDiscrepancyTest / 375
9.5.4 JointTestProcedure / 383
9.5.5 LikelihoodRatioTests / 384
9.5.6 SizeandPoweroftheSymmetricStableTests / 385
9.5.7 ExtensiontoTestingtheAsymmetricStableParetianCase / 395
10 TheMethodofIndirectInference 401
10.1 Introduction / 401
10.2 ApplicationtotheLaplaceDistribution / 403
10.3 ApplicationtoRandomizedResponse / 403
10.3.1 Introduction / 403
10.3.2 EstimationviaIndirectInference / 406
x CONTENTS
10.4 ApplicationtotheStableParetianDistribution / 409
10.5 Problems / 416
A ReviewofFundamentalConceptsinProbabilityTheory 419
A.1 CombinatoricsandSpecialFunctions / 420
A.2 BasicProbabilityandConditioning / 423
A.3 UnivariateRandomVariables / 424
A.4 MultivariateRandomVariables / 427
A.5 ContinuousUnivariateRandomVariables / 430
A.6 ConditionalRandomVariables / 432
A.7 GeneratingFunctionsandInversionFormulas / 434
A.8 ValueatRiskandExpectedShortfall / 437
A.9 JacobianTransformations / 451
A.10 SumsandOtherFunctions / 453
A.11 SaddlepointApproximations / 456
A.12 OrderStatistics / 460
A.13 TheMultivariateNormalDistribution / 462
A.14 NoncentralDistributions / 465
A.15 InequalitiesandConvergence / 467
A.15.1 InequalitiesforRandomVariables / 467
A.15.2 ConvergenceofSequencesofSets / 469
A.15.3 ConvergenceofSequencesofRandomVariables / 473
A.16 TheStableParetianDistribution / 483
A.17 Problems / 492
A.18 Solutions / 509
REFERENCES 537
INDEX 561