Essential Statistical Inference: Theory and Methods

566 Pages·2013·5.512 MB·English

by  Dennis D. Boos, L A Stefanski

Springer Texts in Statistics
SeriesEditors:
G.Casella
S.Fienberg
I.Olkin
Forfurthervolumes:
http://www.springer.com/series/417
Dennis D. Boos • L.A. Stefanski
Essential Statistical Inference
Theory and Methods
123
DennisD.Boos L.A.Stefanski
DepartmentofStatistics DepartmentofStatistics
NorthCarolinaStateUniversity NorthCarolinaStateUniversity
Raleigh,NorthCarolina Raleigh,NorthCarolina
USA USA
ISSN1431-875X
ISBN978-1-4614-4817-4 ISBN978-1-4614-4818-1(eBook)
DOI10.1007/978-1-4614-4818-1
SpringerNewYorkHeidelbergDordrechtLondon
LibraryofCongressControlNumber:2012949516
©SpringerScience+BusinessMediaNewYork2013
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.Exemptedfromthislegalreservationarebriefexcerptsinconnection
with reviews or scholarly analysis or material supplied specifically for the purpose of being entered
and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of
this publication or parts thereof is permitted only under the provisions of the Copyright Law of the
Publisher’slocation,initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer.
PermissionsforusemaybeobtainedthroughRightsLinkattheCopyrightClearanceCenter.Violations
areliabletoprosecutionundertherespectiveCopyrightLaw.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
While the advice and information in this book are believed to be true and accurate at the date of
publication,neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityfor
anyerrorsoromissionsthatmaybemade.Thepublishermakesnowarranty,expressorimplied,with
respecttothematerialcontainedherein.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
To ouradvisors,Bob Serfling(DDB)andRay
CarrollandDavid Ruppert(LAS), who set
ourcareers on trackandgenerouslyand
expertlyguidedusthroughourearly
professionalyears
and
To ourspouses,Kathy(DDB)andDonna
(LAS),whograciouslyandpatientlytolerated
usboththen andnow
Preface
Theoriginsofthisbookdatebacktothe1990swhenweco-taughtasemester-long
advancedinferencecourse.Ourgoalthenandnowistogiveanaccessibleversionof
classicallikelihoodinferenceplusmoderntopicslikeM-estimation,thejackknife,
and the bootstrap. The last chapter on classical permutation and rank methods is
not“modern”butcertainlyistimeless.Ourworldviewisthatmodelsareimportant
forstatisticalframingofscientific questions,buttheM-estimationandresampling
approachesfacilitate robustinferencein the faceof possiblemisspecification.The
Bayesian chapter is a newer addition intended to give a solid introduction to an
increasingly important general approach. Most of the book, however, is clearly
frequentist.
A typical semester course consists of Chaps.1–6 plus selections from Chaps.
7–12.WehavesprinkledRcodethroughoutthetextandalsoinproblems.
We expect students to have taken a first-year graduate level mathematical
statistics course from a text like Statistical Inference by Casella and Berger. But
measure theory is not a requirement and only shows up briefly when discussing
almost sure convergence in Chap.5 and in the Chap.2 appendix on exponential
families.
Althoughintendedforsecond-yeargraduatestudents,manyofthechapterscan
serve as referencesfor researchers. In particular, Chap.9 on Monte Carlo studies,
Chap.10onthejackknife,andChap.12onpermutationmethodscontainresultsand
summariesthatarenoteasilyaccessibleelsewhere.
We thank many generations of students for careful reading and constructive
suggestions.
Raleigh,NC,USA DennisD.Boos
LeonardA.Stefanski
vii
Contents
PartI IntroductoryMaterial
1 RolesofModelinginStatisticalInference ............................... 3
1.1 Introduction......................................................... 3
1.2 ThePartsofaModel................................................ 4
1.3 TheRolesofaModelinDataAnalysis............................ 7
1.4 AConsultingExample.............................................. 8
1.5 Notation,Definitions,Asymptotics,andSimulations............. 10
1.5.1 FamiliesofDistributions ................................ 10
1.5.2 MomentsandCumulants ............................... 11
1.5.3 QuantilesandPercentiles................................ 12
1.5.4 AsymptoticNormalityandSomeBasic
AsymptoticResults...................................... 13
1.5.5 SimulationMethods ..................................... 15
1.6 Example:ASimpleMean/VarianceModel........................ 17
1.6.1 TheAssumedModelandAssociatedEstimators....... 17
1.6.2 Model-BasedComparisonofMOMtoMLE........... 18
1.6.3 ANon-Model-BasedComparisonofMOM
toMLE ................................................... 18
1.6.4 CheckingExampleDetails .............................. 19
1.7 Problems ............................................................ 20
PartII Likelihood-BasedMethods
2 LikelihoodConstructionandEstimation ................................ 27
2.1 Introduction......................................................... 27
2.1.1 NotesandNotation ...................................... 29
2.2 LikelihoodConstruction............................................ 30
2.2.1 DiscreteIIDRandomVariables......................... 30
2.2.2 MultinomialLikelihoods................................ 32
2.2.3 ContinuousIIDRandomVariables ..................... 36
ix
x Contents
2.2.4 MixturesofDiscreteandContinuousComponents.... 41
2.2.5 ProportionalLikelihoods ............................... 42
2.2.6 TheEmpiricalDistributionFunctionasanMLE....... 45
2.2.7 LikelihoodsfromCensoredData........................ 46
2.3 LikelihoodsforRegressionModels................................ 50
2.3.1 LinearModel............................................. 50
2.3.2 AdditiveErrorsNonlinearModel....................... 53
2.3.3 GeneralizedLinearModel............................... 53
2.3.4 GeneralizedLinearMixedModel(GLMM)............ 55
2.3.5 AcceleratedFailureModel .............................. 56
2.4 MarginalandConditionalLikelihoods............................. 57
2.4.1 Neyman-ScottProblem.................................. 57
2.4.2 MarginalLikelihoods.................................... 58
2.4.3 Neyman-ScottProblem via Explicit
ConditionalLikelihood.................................. 59
2.4.4 LogisticRegressionMeasurementErrorModel........ 59
2.4.5 GeneralFormforExponentialFamilies ................ 61
2.4.6 ConditionalLogisticRegression........................ 61
2.5 The Maximum Likelihood Estimator
andtheInformationMatrix......................................... 62
2.5.1 ExamplesofInformationMatrixCalculations ......... 69
2.5.2 VarianceCostforAddingParameterstoaModel...... 75
2.5.3 TheInformationMatrixforTransformed
andModeledParameters ................................ 79
2.6 MethodsforMaximizingtheLikelihoodorSolving
theLikelihoodEquations........................................... 80
2.6.1 AnalyticalMethodsviaProfileLikelihoods............ 81
2.6.2 NewtonMethods......................................... 82
2.6.3 EMAlgorithm ........................................... 83
2.7 AppendixA–UniquenessofMaximumLikelihood
Estimators........................................................... 90
2.7.1 Definitions................................................ 90
2.7.2 MainResults ............................................. 91
2.7.3 ApplicationofTheorem2.2totheMultinomial........ 92
2.7.4 UniquenessoftheMLEintheNormal
location-scaleModel..................................... 93
2.7.5 Applicationof Theorems2.1 and 2.3
totheExponentialThresholdModel.................... 95
2.7.6 UniquenessoftheMLEforExponentialFamilies..... 96
2.8 AppendixB–ExponentialFamilyDistributions.................. 97
2.8.1 CanonicalRepresentation ............................... 99
2.8.2 MinimalExponentialFamily............................ 101
2.8.3 Completeness ............................................ 103
2.8.4 DistributionsoftheSufficientStatistics ................ 104
Contents xi
2.8.5 Familieswith TruncationorThreshold
Parameters ............................................... 105
2.8.6 ExponentialFamilyGlossary ........................... 106
2.9 Problems ............................................................ 107
3 Likelihood-BasedTestsandConfidenceRegions ....................... 125
3.1 SimpleNullHypotheses............................................ 128
3.2 CompositeNullHypotheses........................................ 130
3.2.1 WaldStatistic–Partitioned(cid:2) ........................... 130
3.2.2 ScoreStatistic–Partitioned(cid:2)........................... 131
3.2.3 LikelihoodRatioStatistic–Partitioned(cid:2) .............. 132
3.2.4 NormalLocation-ScaleModel.......................... 132
3.2.5 Wald,Score,andLikelihoodRatioTests–
H Wh.(cid:2)/D0 ........................................... 135
0
3.2.6 SummaryofWald,Score,andLRTestStatistics ...... 136
3.2.7 ScoreStatisticforMultinomialData.................... 137
3.2.8 WaldTestLackofInvariance ........................... 139
3.2.9 TestingEqualityofBinomialProbabilities:
IndependentSamples.................................... 142
3.2.10 TestStatisticsfortheBehrens-FisherProblem......... 143
3.3 ConfidenceRegions................................................. 144
3.3.1 Confidence Intervalfor a Binomial
Probability ............................................... 144
3.3.2 ConfidenceIntervalfortheDifference
ofBinomialProbabilities:IndependentSamples....... 145
3.4 LikelihoodTestingforRegressionModels........................ 146
3.4.1 LinearModel............................................. 146
3.4.2 AdditiveErrorsNonlinearModel....................... 146
3.4.3 GeneralizedLinearModel............................... 146
3.5 ModelAdequacy.................................................... 149
3.6 NonstandardHypothesisTestingProblems........................ 150
3.6.1 One-SidedHypothesesandTheirExtensions.......... 151
3.6.2 NullHypothesesontheBoundary
oftheParameterSpace .................................. 153
3.7 Problems ............................................................ 158
4 BayesianInference.......................................................... 163
4.1 Introduction......................................................... 163
4.2 BayesEstimators.................................................... 170
4.3 CredibleIntervals................................................... 171
4.4 ConjugatePriors .................................................... 172
4.5 NoninformativePriors.............................................. 174
4.6 NormalDataExamples............................................. 176
4.6.1 OneNormalSamplewithUnknownMean
andVariance.............................................. 176
4.6.2 TwoNormalSamples.................................... 178
4.6.3 NormalLinearModel.................................... 179
xii Contents
4.7 HierarchicalBayesandEmpiricalBayes.......................... 182
4.7.1 One-WayNormalRandomEffectsModel.............. 183
4.7.2 James-SteinEstimation.................................. 186
4.7.3 Meta-AnalysisApplicationsofHierarchical
andEmpiricalBayes..................................... 187
4.8 MonteCarloEstimationofaPosterior............................. 193
4.8.1 DirectMonteCarloSamplingfromaPosterior........ 194
4.8.2 MarkovchainMonteCarloSamplingfrom
aPosterior................................................ 195
4.8.3 WhyDoesGibbsSamplingWork?...................... 199
4.9 Problems ............................................................ 201
PartIII LargeSampleApproximationsinStatistics
5 LargeSampleTheory:TheBasics........................................ 207
5.1 Overview............................................................ 207
5.1.1 StatisticsApproximatedbyAverages................... 208
5.2 TypesofStochasticConvergence.................................. 212
5.2.1 ConvergencewithProbability1(Almost
SureConvergence)....................................... 213
5.2.2 ConvergenceinProbability ............................. 214
5.2.3 ConvergenceinDistribution............................. 217
5.3 RelationshipsbetweenTypesofConvergence..................... 223
5.4 ExtensionofConvergenceDefinitionstoVectors................. 223
5.4.1 Convergencewp1andinProbability
forRandomVectors...................................... 224
5.4.2 ConvergenceinDistributionforVectors................ 224
5.5 ToolsforProvingLargeSampleResults........................... 226
5.5.1 MarkovInequality ....................................... 226
5.5.2 ContinuousFunctionsofConvergentSequences....... 227
5.5.3 OrderNotation .......................................... 231
5.5.4 AsymptoticNormalityandRelatedResults ............ 235
5.5.5 TheDeltaTheorem...................................... 237
5.5.6 Slutsky’sTheorem ...................................... 241
5.5.7 ApproximationbyAverages............................. 242
5.5.8 FindinghintheApproximationbyAverages.......... 244
5.5.9 ProvingConvergencein Distribution
ofRandomVectors....................................... 255
5.5.10 MultivariateApproximationbyAverages .............. 257
5.6 SummaryofMethodsforProvingConvergence
inDistribution....................................................... 258
5.7 Appendix–CentralLimitTheoremforIndependent
Non-IdenticallyDistributedSummands ........................... 259
5.7.1 DoubleArrays............................................ 261
5.7.2 Lindeberg-FellerCentralLimitTheorem............... 262
5.8 Problems ............................................................ 263
Contents xiii
6 LargeSampleResultsforLikelihood-BasedMethods.................. 275
6.1 Introduction......................................................... 275
b
6.2 ApproachestoProvingConsistencyof(cid:2) ...................... 275
MLE
6.3 ExistenceofaConsistentRootoftheLikelihoodEquations..... 277
6.3.1 Real-Valued(cid:2) ............................................ 278
6.3.2 Vector(cid:2) .................................................. 280
6.4 CompactParameterSpaces......................................... 282
6.5 AsymptoticNormalityofMaximumLikelihoodEstimators ..... 283
6.5.1 Real-Valued(cid:2) ............................................ 284
6.5.2 Vector(cid:2) .................................................. 286
6.6 AsymptoticNullDistributionofLikelihood-BasedTests......... 287
6.6.1 WaldTests................................................ 287
6.6.2 ScoreTests ............................................... 288
6.6.3 LikelihoodRatioTests................................... 288
6.6.4 LocalAsymptoticPower ................................ 290
6.6.5 NonstandardSituations.................................. 291
6.7 Problems ............................................................ 292
PartIV MethodsforMisspecifiedLikelihoodsandPartially
SpecifiedModels
7 M-Estimation(EstimatingEquations) ................................... 297
7.1 Introduction......................................................... 297
7.2 TheBasicApproach................................................ 300
7.2.1 EstimatorsforA,B,andV ............................. 301
7.2.2 SampleMeanandVariance.............................. 302
7.2.3 RatioEstimator .......................................... 304
7.2.4 DeltaMethodViaM-Estimation........................ 305
7.2.5 PosteriorMode........................................... 306
7.2.6 InstrumentalVariableEstimation ....................... 306
7.3 ConnectionstotheInfluenceCurve(Approximation
byAverages) ........................................................ 309
7.4 Nonsmooth Functions............................................ 311
7.4.1 RobustLocationEstimation............................. 311
7.4.2 QuantileEstimation...................................... 312
7.4.3 PositiveMeanDeviation................................. 313
7.5 RegressionM-Estimators........................................... 314
7.5.1 LinearModelwithRandomX.......................... 314
7.5.2 LinearModelforNonrandomX........................ 315
7.5.3 Nonlinear Model for Nonrandom
X—ExtendedDefinitionsofAandB ................. 318
7.5.4 Robustregression........................................ 320
7.5.5 GeneralizedLinearModels.............................. 321
7.5.6 GeneralizedEstimatingEquations(GEE).............. 322
7.6 ApplicationtoaTestingProblem .................................. 323