Mathematical Statistics: A Unified Introduction

478 Pages·2010·2.369 MB·English

by  George R. Terrell

Mathematical Statistics:
A Unified Introduction
George R. Terrell
Springer

Teacher’s Preface
Why another textbook? The statistical community generally agrees that at the
upper undergraduate level, or the beginning master’s level, students of statistics
shouldbegintostudythemathematicalmethodsofthefield.Weassumethatby
thentheywillhavestudiedtheusualtwo-yearcollegesequence,includingcalculus
through multiple integrals and the basics of matrix algebra. Therefore, they are
readytolearnthefoundationsoftheirsubject,inmuchmoredepththanisusual
inanapplied,“cookbook,”introductiontostatisticalmethodology.
There are a number of well-written, widely used textbooks for such a course.
Theseseemtoreflectaconsensusforwhatneedstobetaughtandhowitshould
betaught.So,whydoweneedyetanotherbookforthisspotinthecurriculum?
Ilearnedmathematicalstatisticswiththehelpofthestandardtexts.Sincethen,
Ihavetaughtthiscourseandsimilaronesmanytimes,atseveraldifferentuniversi-
ties,usingwell-thought-oftextbooks.Butfromthebeginning,Ifeltthatsomething
waswrong.Ittookmeseveralyearstoarticulatetheproblem,andmanymoreto
assemblemysolutionintothebookyouhaveinyourhand.
You see, I spend the rest of my day in statistical consulting and statistical re-
search.Ishouldhavebeenpreparingmymathematicalstatisticsstudentstojoin
meinthisexcitingwork.Butfromseeingwhatthebettergraduatingseniorsand
beginninggraduatestudentsusuallyknew,Iconcludedthatthestandardcurricu-
lumwasnotteachingthemtobesophisticatedcitizensofthestatisticalcommunity.
Theseablestudentsseemedtobewellinformedaboutasetofnarrow,technical
issuesandatthesametimeembarrassinglylackinginanyunderstandingofmore
fundamental matters. For example, many of them could discourse learnedly on
which sources of variation were testable in complicated linear models. But they
becametongue-tiedwhenaskedtoexplain,inEnglish,whatthepresenceofsome
interactionmeantforthereal-worldexperimentunderdiscussion!
vi Teacher’sPreface
Whatwentwrong?Ihavecometobelievethattheproblemliesinourhistory.
The first modern textbooks were written in the 1950s. This was at the end of
the Heroic Age of statistics, roughly, the first half of the twentieth century. Two
bodiesofmagnificentachievementsmarkthatera.Thefirst,identifiedwithStudent,
Fisher,Neyman,Pearson,andmanyothers,developedthephilosophyandformal
methodology of what we now call classical inference. The analysis of scientific
experimentsbecamesostraightforwardthatthesetechniquesswepttheworldof
applications. Many of our clients today seem to believe that these methods are
statistics.
Thesecond,associatedwithLiapunov,Kolmogorov,andmanyothers,wasthe
formalmathematicizationofprobabilityandstatistics.Theseresearchersproved
precisecentrallimittheorems,stronglawsoflargenumbers,andlawsoftheiterated
logarithm(letmecalltheseadvancedasymptotics).Theyaxiomatizedprobability
theory and placed distribution theory on a rigorous foundation, using Lebesgue
integrationandmeasuretheory.
By the 1950s, statisticians were dazzled by these achievements, and to some
extent we still are. The standard textbooks of mathematical statistics show it.
Unfortunately, this causes problems for teachers. Measure theory and advanced
asymptoticsarestillwellbeyondthesophisticationofmostundergraduates,sowe
cannotreallyteachthematthislevel.Furthermore,toomuchclassicalinference
leadsustoneglecttheprecedingtwocenturiesofpowerfulbutlessformalmeth-
ods,nottomentionthebroadadvancesofthelast50years:Bayesianinference,
conditionalinference,likelihood-basedinference,andsoforth.
Sothestandardtextbooksstartwithlong,dry,introductionstoabstractprobabil-
ityanddistributiontheory,almostdevoidofstatisticalmotivationsandexamples
(pokerproblems?!).Thenthereisafranticrush,againlargelyunmotivated,tointro-
duceexactlythosedistributionsthatwillbeneededforclassicalinference.Finally,
two-thirdsofthewaythrough,thefirstrealstatisticalapplicationsappear—means
tests,one-wayANOVA,etc.—butrigidlyconfinedwithintheclassicalinferential
framework.(Anearlyreaderofthemanuscriptcalledthis“thecultofthet-test.”)
Finally,inperhapsChapter14,thebooksgettolinearregression.Now,regression
is200yearsold,easy,intuitive,andincrediblyuseful.Unfortunately,ithasbeen
madeverydifficult:“conditioningofmultivariateGaussiandistributions”asone
cultistputit.Fortunately,itappearssolateinthetermthatitgetsomittedanyway.
We distort the details of teaching, too, by our obsession with graduate-level
rigor.Large-sampletheoryisattheheartofstatisticalthinking,butweareafraid
totouchit.“Asymptoticsconsistsofcorollariestothecentrallimittheorem,”as
another cultist puts it. We seem to have forgotten that 200 years of what I shall
callelementaryasymptoticsprecededLiapunov’swork.Furthermore,thefearof
sayinganythingthatwillhavetobemodifiedlater(ingraduateclassesthatassume
measuretheory)forcesundergraduatemathematicalstatisticstextstoincludevery
littlerealmathematics.
Asaresult,mostofthesestandardtextsarehardlydifferentfromthecookbooks,
withafewintegralstossedinforflavor,likejalapen˜obitsincornbread.Othersare
spicedwithdefinitionsandtheoremshedgedaboutwithverytechnicalconditions,
Teacher’sPreface vii
which are never motivated, explained, or applied (remember “regularity condi-
tions”?).Mathematicalproofs,surelyabasictoolforunderstanding,areconfined
to a scattering of places, chosen apparently because the arguments are easy and
“elegant.”Elsewhere,thedemoralizingrefrainbecomes“theproofisbeyondthe
scopeofthiscourse.”
Howisthisbookdifferent?Inshort,thisbookisintendedtoteachstudentsto
domathematicalstatistics,notjusttoappreciateit.Therefore,Ihaveredesignedthe
coursefromfirstprinciples.Ifyouarefamiliarwithastandardtextbookonthesub-
jectandyouopenthisoneatrandom,youareverylikelytofindeitherasurprising
topicoranunexpectedtreatmentorplacementofastandardtopic.Buteverything
ishereforareason,anditsorderofappearancehasbeencarefullychosen.
First, as the subtitle implies, the treatment in unified. You will find here no
artificial separation of probability from statistics, distribution theory from infer-
ence,orestimationfromhypothesistesting.Itreatprobabilityasamathematical
handmaiden of statistics. It is developed, carefully, as it is needed. A statistical
motivationforeachaspectofprobabilitytheoryisthereforeprovided.
Second, I have updated the range of subjects covered. You will encounter in-
troductionstosuchimportantmoderntopicsasloglinearmodelsforcontingency
tables and logistic regression models (very early in the book!), finite population
sampling,branchingprocesses,andsmall-sampleasymptotics.
More important are the matters I emphasize systematically. Asymptotics is a
major theme of this book. Many large-sample results are not difficult and quite
appropriatetoanundergraduatecourse.Forexample,Ihadalwaystaughtthatwith
“largen,smallp”onemayusethePoissonapproximationtobinomialprobabil-
ities. Then I would be embarrassed when a student asked me exactly when this
worked.Sowederivehereasimple,usefulerrorboundthatanswersthisquestion.
Naturally,afullmoderncentrallimittheoremismathematicallyabovethelevelof
thiscourse.Butagreatnumberofusefulyetmoreelementarynormallimitresults
exist,andmanyarederivedhere.
I emphasize those methods and concepts that are most useful in statistics in
the broad sense. For example, distribution theory is motivated by detailed study
ofthemostwidelyusefulfamiliesofrandomvariables.Classicalestimationand
hypothesis testing are still dealt with, but as applications of these general tools.
Simultaneously,Bayesian,conditional,andotherstylesofinferenceareintroduced
aswell.
Thestandardtextbooks,unfortunately,tendtointroduceveryobscureandab-
stractsubjects“cold”(wheredidahorribleexpressionlike√1 e−x2/2comefrom?),
2π
thenonlybelatedlygetaroundtomotivatingthemandgivingexamples.Herewe
insistonconcreteness.Thebookprecedeseachnewtopicwitharelevantstatistical
problem. We introduce abstract concepts gradually, working from the special to
thegeneral.Atthesametime,eachnewtechniqueisappliedaswidelyaspossible.
Thus, every chapter is quite broad, touching on many connections with its main
topics.
Thebook’sattitudetowardmathematicsmaysurpriseyou:Wetakeitseriously.
Ourstudentsmaynotknowmeasuretheory,buttheydoknowanenormousamount
viii Teacher’sPreface
ofusefulmathematics.Thistextuseswhattheydoknowandteachesthemmore.
We aim for reasonable completeness: Every formula is derived, every property
is proved (often, students are asked to complete the arguments themselves as
exercises).Thelevelofmathematicalprecisionandgeneralityisappropriatetoa
seriousupper-levelundergraduatecourse.
Atthesametime,studentsarenotexpectedtomemorizeexotictechnicalities,
relevantonlyingraduateschool.Forexample,thebookdoesnotburdenthemwith
theinfamous“triple”definitionofarandomvariable;alessobscuredefinitionis
adequateforourworkhere.(Thosestudentswhogoontograduatemathematical
statistics courses will be just the ones who will have no trouble switching to
themoreabstractpointofviewlater.)Furthermore,weemphasizemathematical
directness: Those short, elegant proofs so prized by professors are often here
replacedbyslightlylongerbutmoreconstructivedemonstrations.Ourgoalisto
stimulateunderstanding,nottodazzlewithourbrilliance.
Whatisinthebook?Thesepedagogicalprinciplesimposeanunconventional
orderoftopics.Letmetakeyouonabrieftourofthebook:
The “Getting Started” chapter motivates the study of statistics, then prepares
thestudentforhands-oninvolvement:completingproofsandderivationsaswell
asworkingproblems.
Chapter 1 adopts an attitude right away: Statistics precedes probability. That
is, models for important phenomena are more important than models for mea-
surement and sampling error. The first two chapters do not mention probability.
We start with the linear data-summary models that make up so much of statisti-
calpractice:one-waylayoutsandfactorialmodels.Fundamentalconceptssuchas
additivityandinteractionappearnaturally.Thesimplestlinearregressionmodels
followbyinterpolation.Thenweconstructsimplecontingency-tablemodelsfor
counting experiments and thereby discover independence and association. Then
wetakelogarithms,toderiveloglinearmodelsforcontingencytables(whichare
strikingly parallel to our linear models). Again, logistic regression models arise
byinterpolation.Inthischapter,ofcourse,werestrictourselvestocasesforwhich
reasonableparameterestimatesareobvious.
Chapter2showshowtoestimateANOVAandregressionmodelsbytheancient,
intuitivemethodofleastsquares.Weemphasizegeometricalinterpolationofthe
method—shortestEuclideandistance.Thismotivatessamplevariance,covariance,
andcorrelation.DecompositionofthesumofsquaresinANOVAandinsightinto
degreesoffreedomfollownaturally.
That is as far as we can go without models for errors, so Chapter 3 begins
withaconventionalintroductiontocombinatorialprobability.Itis,however,very
concrete: We draw marbles from urns. Rather than treat conditional probability
as a later, artificially difficult topic, we start with the obvious: All probabilities
are conditional. It is just that a few of them are conditional on a whole sample
space.Thenthefirstasymptoticresultisobtained,toaidintheunderstandingof
thefamous“birthdayproblem.”Thisleadstoinsightintothedifferencebetween
finitepopulationandinfinitepopulationsampling.
Teacher’sPreface ix
Chapter4usesgeometricalexamplestointroducecontinuousprobabilitymod-
els.Thenwegeneralizetoabstractprobability.Theaxiomsweusecorrespondto
howoneactuallycalculatesprobability.Wegoontogeneraldiscreteprobability,
andBayes’stheorem.ThechapterendswithanelementaryintroductiontoBorel
algebraasabasisforcontinuousprobabilities.
Chapter 5 introduces discrete random variables. We start with finite popula-
tion sampling, in particular, the negative hypergeometric family. You may not
be familiar with this family, but the reasons to be interested are numerous: (1)
Manycommonrandomvariables(binomial,negativebinomial,Poisson,uniform,
gamma, beta, and normal) are asymptotic limits of this family; (2) it possesses
intransparentwaysthesymmetriesanddualitiesofthosefamilies;and(3)itbe-
comesparticularlyeasyforthestudenttocarryouthisownsimulations,viaurn
models.ThentheFisherexacttestgivesusthefirstexampleofanhypothesistest,
for independence in the 2×2 tables we studied in Chapter 1. We introduce the
expectation of discrete random variables as a generalization of the average of a
finitepopulation.Finally,wegivethefirstestimatesforunknownparametersand
confidenceboundsforthem.
Chapter6introducesthegeometric,negativebinomial,binomial,andPoisson
families.Wediscoverthatthefirstthreeariseasasymptoticlimitsinthenegative
hypergeometric family and also as sequences of Bernoulli experiments. Thus,
wehaverelatedfiniteandinfinitepopulationsampling.Weinvestigatejustwhen
thePoissonfamilymaybeusedasanasymptoticapproximationinthebinomial
andnegativebinomialfamilies.Generaldiscreteexpectationsandthepopulation
variancearethenintroduced.Confidenceintervalsandtwo-sidedhypothesistests
providenaturalapplications.
Chapter 7 introduces random vectors and random samples. Here is where
marginalandconditionaldistributionsappear,andfromthese,populationcovari-
anceandcorrelation.Thistellsussomethingsaboutthedistributionofthesample
meanandvariance,andleadstothefirstlawsoflargenumbers.Thestudyofcon-
ditionaldistributionspermitsthefirstexamplesofparametricBayesianinference.
Chapter8investigatesparameterestimationandevaluationoffitincomplicated
discretemodels.Weintroducethediscretelikelihoodandthelog-likelihoodratio
statistic. This turns out often to be asymptotically equivalent to Pearson’s chi-
squaredstatistic,butitismuchmoregenerallyuseful.Thenweintroducemaximum
likelihoodestimationandapplyittologlinearcontingencytablemodels;estimates
arecomputedbyiterativeproportionalfitting.Weestimatelinearlogisticmodels
bymaximumlikelihood,evaluatedbyNewton’smethod.
Chapter 9 constructs the Poisson process, from which we obtain the gamma
family.ThenaDirichletprocessisconstructed,fromwhichwegetthebetafamily.
Connectionsbetweenthesetwofamiliesareexplored.Thecontinuousversionof
thelikelihoodratioisintroduced,andweuseittoestablishtheNeyman–Pearson
lemma.
Chapter10definesthegeneralquantilefunctionofarandomvariable,byasking
how we might simulate it. Then we may define the expectation of any random