Toprotecttherightsoftheauthor(s)andpublisherweinformyouthatthisPDFisanuncorrectedproofforinternalbusinessuseonlybytheauthor(s),editor(s), reviewer(s),ElsevierandtypesetterdiacriTech.Itisnotallowedtopublishthisproofonlineorinprint.Thisproofcopyisthecopyrightpropertyofthepublisher andisconfidentialuntilformalpublication. WILCOX FM-9780123869838 2011/10/13 15:19 Page 1 #1 Introduction to Robust Estimation and Hypothesis Testing Toprotecttherightsoftheauthor(s)andpublisherweinformyouthatthisPDFisanuncorrectedproofforinternalbusinessuseonlybytheauthor(s),editor(s), reviewer(s),ElsevierandtypesetterdiacriTech.Itisnotallowedtopublishthisproofonlineorinprint.Thisproofcopyisthecopyrightpropertyofthepublisher andisconfidentialuntilformalpublication. WILCOX 01-fm-i-iv-9780123869838 2011/12/6 18:37 Page 3 #3 Introduction to Robust Estimation and Hypothesis Testing 3rd Edition Rand Wilcox AMSTERDAM•BOSTON•HEIDELBERG•LONDON NEWYORK•OXFORD•PARIS•SANDIEGO SANFRANCISCO•SINGAPORE•SYDNEY•TOKYO AcademicPressisanimprintofElsevier Toprotecttherightsoftheauthor(s)andpublisherweinformyouthatthisPDFisanuncorrectedproofforinternalbusinessuseonlybytheauthor(s),editor(s), reviewer(s),ElsevierandtypesetterdiacriTech.Itisnotallowedtopublishthisproofonlineorinprint.Thisproofcopyisthecopyrightpropertyofthepublisher andisconfidentialuntilformalpublication. WILCOX FM-9780123869838 2011/11/10 12:06 Page 4 #4 AcademicPressisanimprintofElsevier 225WymanStreet,Waltham,MA02451,USA 525BStreet,Suite1900,SanDiego,CA92101-4495,USA TheBoulevard,LangfordLane,Kidlington,OxfordOX51GB,UK Radarweg29,POBox211,1000AEAmsterdam,TheNetherlands Firstedition2012 Copyright©2012ElsevierInc.Allrightsreserved. Nopartofthispublicationmaybereproduced,storedinaretrievalsystemortransmittedinanyformorbyanymeans electronic,mechanical,photocopying,recordingorotherwisewithoutthepriorwrittenpermissionofthepublisher PermissionsmaybesoughtdirectlyfromElsevier’sScience&TechnologyRightsDepartmentinOxford,UK:phone (+44)(0)1865843830;fax(+44)(0)1865853333;email:[email protected] requestonlinebyvisitingtheElsevierwebsiteathttp://elsevier.com/locate/permissions,andselectingObtaining permissiontouseElseviermaterial. Notice Noresponsibilityisassumedbythepublisherforanyinjuryand/ordamagetopersonsorpropertyasamatterofproducts liability,negligenceorotherwise,orfromanyuseoroperationofanymethods,products,instructionsorideascontained inthematerialherein.Becauseofrapidadvancesinthemedicalsciences,inparticular,independentverificationof diagnosesanddrugdosagesshouldbemade. LibraryofCongressCataloging-in-PublicationData AcatalogrecordforthisbookisavailablefromtheLibraryofCongress. Applicationsubmitted. BritishLibraryCataloguing-in-PublicationData AcataloguerecordforthisbookisavailablefromtheBritishLibrary. ISBN:978-0-12-386983-8 ForinformationonallAcademicPresspublications visitourwebsiteatwww.elsevierdirect.com Typesetby:diacriTech,India PrintedandboundinUSA 12 13 14 15 10 9 8 7 6 5 4 3 2 1 Toprotecttherightsoftheauthor(s)andpublisherweinformyouthatthisPDFisanuncorrectedproofforinternalbusinessuseonlybytheauthor(s),editor(s), reviewer(s),ElsevierandtypesetterdiacriTech.Itisnotallowedtopublishthisproofonlineorinprint.Thisproofcopyisthecopyrightpropertyofthepublisher andisconfidentialuntilformalpublication. WILCOX preface-9780123869838 2011/10/25 1:23 Page xix #1 Preface Thisbookfocusesonthepracticalaspectsofmodernandrobuststatisticalmethods.The increasedaccuracyandpowerofmodernmethods,versusconventionalapproachestothe analysisofvariance(ANOVA)andregression,isremarkable.Throughacombinationof theoreticaldevelopments,improvedandmoreflexiblestatisticalmethods,andthepowerof thecomputer,itisnowpossibletoaddressproblemswithstandardmethodsthatseemed insurmountableonlyafewyearsago. Themostcommonapproachwhencomparingtwoormoregroupsistocomparemeans, assumingthatobservationshavenormaldistributions.Whencomparingindependentgroups, itisfurtherassumedthatdistributionshaveacommonvariance.Conventionalwisdomisthat thesestandardANOVAmethodsarerobusttoviolationsofassumptions.Thisviewisbasedin largepartonstudies,publishedbeforetheyear1960,showingthatifgroupsdonotdiffer (meaningthattheyhaveidenticaldistributions),thengoodcontrolovertheprobabilityofa typeIerrorisachieved.However,ifgroupsdiffer,hundredsofmorerecentjournalarticles havedescribedseriouspracticalproblemswithstandardtechniquesandhowtheseproblems mightbeaddressed.Oneconcernisthatthesamplemeancanhavearelativelylargestandard errorunderslightdeparturesfromnormality.Thisinturncanmeanlowpower.Another problemisthatprobabilitycoverage,basedonconventionalmethodsforconstructing confidenceintervals,canbesubstantiallydifferentfromthenominallevel,andundesirable powerpropertiesariseaswell.Inparticular,powercangodownasthedifferencebetweenthe meansgetslarge.Theresultisthatimportantdifferencesbetweengroupsareoftenmissed, andthemagnitudeofthedifferenceispoorlycharacterized.Putanotherway,groupsprobably differwhennullhypothesesarerejectedwithstandardmethods,butinmanysituations, standardmethodsaretheleastlikelytofindadifference,andtheyofferapoorsummaryof howgroupsdifferandthemagnitudeofthedifference.Yetanotherfundamentalconcernis thatthepopulationmeanandvariancearenotrobust,roughlymeanthatunderarbitrarily smallshiftsfromnormality,theirvaluescanbesubstantiallyalteredandpotentiallymislead. Thus,evenwitharbitrarilylargesamplesizes,thesamplemeanandvariancemightprovide anunsatisfactorysummaryofthedata. Whendealingwithregression,thesituationisevenworse.Thatis,thereareevenmoreways inwhichanalyses,basedonconventionalassumptions,canbemisleading.Thevery xix Toprotecttherightsoftheauthor(s)andpublisherweinformyouthatthisPDFisanuncorrectedproofforinternalbusinessuseonlybytheauthor(s),editor(s), reviewer(s),ElsevierandtypesetterdiacriTech.Itisnotallowedtopublishthisproofonlineorinprint.Thisproofcopyisthecopyrightpropertyofthepublisher andisconfidentialuntilformalpublication. WILCOX preface-9780123869838 2011/10/25 1:23 Page xx #2 Preface foundationofstandardregressionmethods,namelyestimationviatheleastsquaresprinciple, leadstopracticalproblems,asdoviolationsofotherstandardassumptions.Forexample,if theerrorterminthestandardlinearmodelhasanormaldistribution,butisheteroscedastic, theleastsquaresestimatorcanbehighlyinefficient,andtheconventionalconfidenceinterval fortheregressionparameterscanbeextremelyinaccurate. In1960,itwasunclearhowtoformallydevelopsolutionstothemanyproblemsthathave beenidentified.ItwasthetheoryofrobustnessdevelopedbyP.HuberandF.Hampelthat pavedtheroadforfindingpracticalsolutions.Today,therearemanyasymptoticallycorrect waysofsubstantiallyimprovingonstandardANOVAandregressionmethods.Thatis,they convergetothecorrectanswerasthesamplesizesgetlarge,butsimulationstudieshave shownthatwhensamplesizesaresmall,notallmethodsshouldbeused.Moreover,formany methods,itremainsunclearhowlargethesamplesizesmustbebeforereasonablyaccurate resultsareobtained.Oneofthegoalsofthisbookistoidentifythosemethodsthatperform wellinsimulationstudies,aswellasthosethatdonot. Thisbookdoesnotprovideanencyclopedicdescriptionofalltherobustmethodsthatmight beused.Althoughsomemethodsareexcludedbecausetheyperformpoorlyrelativetoothers, manymethodshavenotbeenexaminedinsimulationstudies,sotheirpracticalvalueremains unknown.Indeed,therearesomanymethods,amassiveeffortisneededtoevaluatethem. Moreover,somemethodsaredifficulttostudywithcurrentcomputertechnology.Thatis,they requiresomuchexecutiontimethatsimulationsremainimpractical.Ofcourse,thismight changeinthenearfuture,butwhatisneedednowisadescriptionofmodernrobustmethods thathavepracticalvalueinappliedwork. Althoughthegoalistofocusontheappliedaspectsofrobustmethods,itisimportantto discussthefoundationsofmodernmethods,sothisisdoneinChapters2and3,andtosome extentinChapter4.Onegeneralpointisthatmodernmethodshaveasolidmathematical foundation.Anothergoalistoimpartthegeneralflavorandaimsofrobustmethods.Thisis importantbecausemisconceptionsarerampant.Forexample,someindividualsfirmlybelieve thatoneofthegoalsofmodernrobustmethodsistofindbetterwaysofestimatingµ,the populationmean.Fromarobustpointofview,thisgoalisnotremotelyrelevant,anditis importanttounderstandwhy.Anothermisconceptionisthatrobustmethodsonlyperform wellwhendistributionsaresymmetric.Infact,boththeoryandsimulationsindicatethat robustmethodsofferanadvantageoverstandardmethodswhendistributionsareskewed. Apracticalconcernisapplyingthemethodsdescribedinthisbook.Manyofthe recommendedmethodshavebeendevelopedinonlythelastfewyearsandarenotavailable instandardstatisticalpackagesforthecomputer.Todealwiththisproblem,easy-to-useR functionsaresupplied.(Withthisthirdedition,S-PLUSfunctionsarenolongersupported.) TheycanbeobtainedasindicatedinSection1.8ofChapter1.WithoneRcommand,allof xx Toprotecttherightsoftheauthor(s)andpublisherweinformyouthatthisPDFisanuncorrectedproofforinternalbusinessuseonlybytheauthor(s),editor(s), reviewer(s),ElsevierandtypesetterdiacriTech.Itisnotallowedtopublishthisproofonlineorinprint.Thisproofcopyisthecopyrightpropertyofthepublisher andisconfidentialuntilformalpublication. WILCOX preface-9780123869838 2011/10/25 1:23 Page xxi #3 Preface thefunctionsdescribedinthisbookbecomeapartofyourversionofR.Illustrations,using thesefunctions,areincluded. Thebookassumesthatthereaderhashadanintroductorystatisticscourse.Thatis,allthatis requiredissomeknowledgeaboutthebasicsofANOVA,hypothesistesting,andregression. Thefoundationsofrobustmethods,describedinChapter2,arewrittenatarelatively nontechnicallevel,buttheexpositionismuchmoretechnicalthantherestofthebook,andit mightbetootechnicalforsomereaders.ItisrecommendedthatChapter2bereadoratleast skimmed,butthosewhoarewillingtoacceptcertainresultscanskiptoChapter3.Oneofthe mainpointsinChapter2isthattherobustmeasuresoflocationandscalethatareusedarenot arbitrary,butwerechosentosatisfyspecificcriteria.Moreover,thesecriteriaeliminatefrom considerationthepopulationmean,variance,andtheusualcorrelationcoefficient. Fromanappliedpointofview,Chapters4–11,whichincludemethodsforaddressing commonproblemsinANOVAandregression,formtheheartofthebook.Technicaldetails arekepttoaminimum.Thegoalistoprovideasimpledescriptionofthebestmethods available,basedontheoreticalandsimulationstudies,andtoprovideadviceonwhich methodstouse.Usually,nosinglemethoddominatesallothers,onereasonbeingthatthere aremultiplecriteriaforjudgingaparticulartechnique.Accordingly,therelativemeritsofthe variousmethodsarediscussed.Althoughnosinglemethoddominates,standardmethodsare typicallytheleastsatisfactory,andmanyalternativemethodscanbeeliminated. xxi Toprotecttherightsoftheauthor(s)andpublisherweinformyouthatthisPDFisanuncorrectedproofforinternalbusinessuseonlybytheauthor(s),editor(s), reviewer(s),ElsevierandtypesetterdiacriTech.Itisnotallowedtopublishthisproofonlineorinprint.Thisproofcopyisthecopyrightpropertyofthepublisher andisconfidentialuntilformalpublication. WILCOX Ch01-9780123869838 2011/10/21 16:54 Page 1 #1 CHAPTER 1 Introduction Introductorystatisticscoursesdescribemethodsforcomputingconfidenceintervalsand testinghypothesesaboutmeansandregressionparametersbasedontheassumptionthat observationsarerandomlysampledfromnormaldistributions.Whencomparingindependent groups,standardmethodsalsoassumethatgroupshaveacommonvariance,evenwhenthe meansareunequal,andasimilarhomogeneityofvarianceassumptionismadewhentesting hypothesesaboutregressionparameters.Currently,thesemethodsformthebackboneofmost appliedresearch.Thereis,however,aseriouspracticalproblem:Manyjournalarticleshave illustratedthatthesestandardmethodscanbehighlyunsatisfactory.Oftentheresultisapoor understandingofhowgroupsdifferandthemagnitudeofthedifference.Powercanbe relativelylowcomparedtorecentlydevelopedmethods,leastsquaresregressioncanyielda highlymisleadingsummaryofhowtwoormorerandomvariablesarerelatedascantheusual correlationcoefficient,theprobabilitycoverageofstandardmethodsforcomputing confidenceintervalscandiffersubstantiallyfromthenominalvalue,andtheusualsample variancecangiveadistortedviewoftheamountofdispersionamongapopulationof participants.Eventhepopulationmean,ifitcouldbedeterminedexactly,cangiveadistorted viewofwhatthetypicalparticipantislike. Althoughtheproblemsjustdescribedarewellknowninthestatisticsliterature,many textbookswrittenfornonstatisticiansstillclaimthatstandardtechniquesarecompletely satisfactory.Consequently,itisimportanttoreviewtheproblemsthatcanariseandwhythese problemsweremissedforsomanyyears.Aswillbecomeevident,severalpiecesof misinformationhavebecomepartofstatisticalfolkloreresultinginafalsesenseofsecurity whenusingstandardstatisticaltechniques. 1.1 Problems with Assuming Normality Tobegin,distributionsarenevernormal.Forsomethisseemsobvious,hardlyworth mentioning,butanaphorismgivenbyCrame´r(1946)andattributedtothemathematician Poincare´ remainsrelevant:“Everyonebelievesinthe[normal]lawoferrors,the experimentersbecausetheythinkitisamathematicaltheorem,themathematiciansbecause theythinkitisanexperimentalfact.”Granted,thenormaldistributionisthemostimportant IntroductiontoRobustEstimationandHypothesisTesting.DOI:10.1016/B978-0-12-386983-8.00001-9 Copyright©2012ElsevierInc.Allrightsreserved. 1 Toprotecttherightsoftheauthor(s)andpublisherweinformyouthatthisPDFisanuncorrectedproofforinternalbusinessuseonlybytheauthor(s),editor(s), reviewer(s),ElsevierandtypesetterdiacriTech.Itisnotallowedtopublishthisproofonlineorinprint.Thisproofcopyisthecopyrightpropertyofthepublisher andisconfidentialuntilformalpublication. WILCOX Ch01-9780123869838 2011/10/21 16:54 Page 2 #2 2 IntroductiontoRobustEstimationandHypothesisTesting distributioninallaspectsofstatistics.Butintermsofapproximatingthedistributionofany continuousdistribution,itcanfailtothepointthatpracticalproblemsarise,aswillbecome evidentatnumerouspointsinthisbook.Tobelieveinthenormaldistributionimpliesthat onlytwonumbersarerequiredtotelluseverythingabouttheprobabilitiesassociatedwitha randomvariable:thepopulationmeanµandpopulationvarianceσ2.Moreover,assuming normalityimpliesthatdistributionsmustbesymmetric. Ofcourse,nonnormalityisnot,byitself,adisaster.Perhapsanormaldistributionprovidesa goodapproximationofmostdistributionsthatariseinpractice,andthereisthecentrallimit theorem,whichtellsusthatunderrandomsampling,asthesamplesizegetslarge,thelimiting distributionofthesamplemeanisnormal.Unfortunately,evenwhenanormaldistribution providesagoodapproximationtotheactualdistributionbeingstudied(asmeasuredbythe Kolmogorovdistancefunctiondescribedlater)practicalproblemsarise.Also,empirical investigationsindicatethatdeparturesfromnormality,thathavepracticalimportance,are rathercommoninappliedwork(e.g.,Hill&Dixon,1982;Micceri,1989;Wilcox,2009a). Evenoveracenturyago,KarlPearsonandotherresearcherswereconcernedaboutthe assumptionthatobservationsfollowanormaldistribution(e.g.,Hand,1998,p.649).In particular,distributionscanbehighlyskewed,theycanhaveheavytails(tailsthatarethicker thananormaldistribution),andrandomsamplesoftenhaveoutliers(unusuallylargeorsmall valuesamongasampleofobservations).Outliersandheavy-taileddistributionsareserious practicalproblemsbecausetheyinflatethestandarderrorofthesamplemean,sopowercan berelativelylowwhencomparinggroups.Modernrobustmethodsprovideaneffectiveway ofdealingwiththisproblem.Fisher(1922),forexample,wasawarethatthesamplemean couldbeinefficientunderslightdeparturesfromnormality. Aclassicwayofillustratingtheeffectsofslightdeparturesfromnormalityiswiththe contaminated ormixednormaldistribution(Tukey,1960).Let X beastandardnormal randomvariablehavingdistribution(cid:56)(x)= P(X ≤x).Thenforanyconstant K >0, (cid:56)(x/K)isanormaldistributionwithstandarddeviation K.Let(cid:15) beanyconstant,0≤(cid:15) ≤1. Thecontaminatednormaldistributionis H(x)=(1−(cid:15))(cid:56)(x)+(cid:15)(cid:56)(x/K), (1.1) whichhasmean0andvariance1−(cid:15)+(cid:15)K2.(Stigler,1973,findsthattheuseofthe contaminatednormaldatesbackatleasttoNewcomb,1896.)Inotherwords,thecontaminated normalarisesbysamplingfromastandardnormaldistributionwithprobability1−(cid:15); otherwise,samplingisfromanormaldistributionwithmean0andstandarddeviation K. Toprovideamoreconcreteexample,considerthepopulationofalladults,andsupposethat 10%ofalladultsareatleast70yearsold.Ofcourse,individualsatleast70yearsoldmight haveadifferentdistributionfromtherestofthepopulation.Forinstance,individualsunder theageof70mighthaveastandardnormaldistribution,butindividualsatleast70yearsold www.elsevierdirect.com Toprotecttherightsoftheauthor(s)andpublisherweinformyouthatthisPDFisanuncorrectedproofforinternalbusinessuseonlybytheauthor(s),editor(s), reviewer(s),ElsevierandtypesetterdiacriTech.Itisnotallowedtopublishthisproofonlineorinprint.Thisproofcopyisthecopyrightpropertyofthepublisher andisconfidentialuntilformalpublication. WILCOX Ch01-9780123869838 2011/10/21 16:54 Page 3 #3 Chapter1 Introduction 3 mighthaveanormaldistributionwithmean0andstandarddeviation10.Then,theentire populationofadultshasacontaminatednormaldistributionwith(cid:15) =.1and K =10.In symbols,theresultingdistributionis H(x)=0.9(cid:56)(x)+0.1(cid:56)(x/10), (1.2) whichhasmean0andvariance10.9.Moreover,Eq.(1.2)isnotanormaldistribution, verificationofwhichisleftasanexercise. To illustrate problems that arise under slight departures from normality, we first examine Eq. (1.2) more closely. Figure 1.1 shows the standard normal and the contaminated normal probability density function corresponding to Eq. (1.2). Notice that the tails of the contaminated normal are above the tails of the normal, so the contaminated normal is said to have heavy tails. It might seem that the normal distribution provides a good approximation of the contaminated normal, but there is an important difference. The standard normal has variance 1, but the contaminated normal has variance 10.9. The reason for the seemingly large difference between the variances is that σ2 is very sensitive to the tails of a distribution. In essence, a small proportion of the population of participants can have an inordinately large effect on its value. Put another way, even when the variance is 0.4 Normal curve 0.3 0.2 0.1 0.0 −3 −2 −1 0 1 2 3 x Figure1.1: Normalandcontaminatednormaldistributions. www.elsevierdirect.com Toprotecttherightsoftheauthor(s)andpublisherweinformyouthatthisPDFisanuncorrectedproofforinternalbusinessuseonlybytheauthor(s),editor(s), reviewer(s),ElsevierandtypesetterdiacriTech.Itisnotallowedtopublishthisproofonlineorinprint.Thisproofcopyisthecopyrightpropertyofthepublisher andisconfidentialuntilformalpublication. WILCOX Ch01-9780123869838 2011/10/21 16:54 Page 4 #4 4 IntroductiontoRobustEstimationandHypothesisTesting known, if sampling is from the contaminated normal, the length of the standard confidence interval for the population mean, µ, will be over three times longer than it would be when sampling from the standard normal distribution instead. What is important from a practical point of view is that there are location estimators other than the sample mean that have standard errors that are substantially less affected by heavy tailed distributions. By “measure of location,” it is meant that some measure intended to represent the typical participant or object, the two best-known examples being the mean and the median. (A more formal definition is given in Chapter 2.) Some of these measures have relatively short confidence intervals when distributions have a heavy tail, yet the length of the confidence interval remains reasonably short when sampling from a normal distribution instead. Put another way, there are methods for testing hypotheses that have good power under normality, but that continue to have good power when distributions are nonnormal, in contrast to methods based on means. For example, when sampling from the contaminated normal given by Eq. (1.2), both Welch’s and Student’s method for comparing the means of two independent groups have power approximately 0.278 when testing at the 0.05 level with equal sample sizes of 25 and when the difference between the means is 1. In contrast, several other methods, described in Chapter 5, have power exceeding 0.7. Inanattempttosalvagethesamplemean,itmightbearguedthatinsomesensethe contaminatednormalrepresentsanextremedeparturefromnormality.Theextremequantiles ofthetwodistributionsdodiffersubstantially,butbasedonvariousmeasuresofthedifference betweentwodistributions,theyareverysimilarassuggestedbyFigure1.1.Forexample,the Kolmogorovdistancebetweenanytwodistributions, F andG,isthemaximumvalueof (cid:49)(x)=|F(x)−G(x)|, themaximumbeingtakenoverallpossiblevaluesof x.(Ifthemaximumdoesnotexist,the supremumorleastupperboundisused.)Ifdistributionsareidentical,theKolmogorov distanceis0,anditsmaximumpossiblevalueis1,asisevident.Nowconsiderthe Kolmogorovdistancebetweenthecontaminatednormaldistribution, H(x),givenby(1.2), andthestandardnormaldistribution,(cid:56)(x).Itcanbeseenthat(cid:49)(x)doesnotexceed0.04for any x.Thatis,basedonaKolmogorovdistancefunction,thetwodistributionsaresimilar. Severalalternativemethodsareoftenusedtomeasurethedifferencebetweendistributions. (SomeofthesearediscussedbyHuberandRonchetti,2009.)Thechoiceamongthese measuresisofinterestwhendealingwiththeoreticalissues,buttheseissuesgobeyondthe scopeofthisbook.Sufficeittosaythatthedifferencebetweenthenormalandcontaminated normalisagainsmall.Gleason(1993)discussesthedifferencebetweenthenormaland contaminatednormalfromadifferentperspectiveandalsoconcludesthatthedifferenceis small. Evenifitcouldbeconcludedthatthecontaminatednormalrepresentsalargedeparture fromnormality,concernsoverthesamplemeanwouldpersist,forreasonsalreadygiven. www.elsevierdirect.com
Description: