Model-Free Prediction and Regression: A Transformation-Based Approach to Inference PDF
Preview Model-Free Prediction and Regression: A Transformation-Based Approach to Inference
Frontiers in Probability and the Statistical Sciences Dimitris N. Politis Model-Free Prediction and Regression A Transformation-Based Approach to Inference Frontiers in Probability and the Statistical Sciences Editor-inChief: SomnathDatta DepartmentofBioinformatics&Biostatistics UniversityofLouisville Louisville,Kentucky,USA SeriesEditors: FrederiG.Viens DepartmentofMathematics&DepartmentofStatistics PurdueUniversity WestLafayette,Indiana,USA DimitrisN.Politis DepartmentofMathematics UniversityofCalifornia,SanDiego LaJolla,California,USA HannuOja DepartmentofMathematicsandStatistics UniversityofTurku Turku,Finland MichaelDaniels SectionofIntegrativeBiology DivisionofStatistics&ScientificComputation UniversityofTexas Austin,Texas,USA Moreinformationaboutthisseriesathttp://www.springer.com/series/11957 Dimitris N. Politis Model-Free Prediction and Regression A Transformation-Based Approach to Inference 123 DimitrisN.Politis DepartmentofMathematics UniversityofCalifornia,SanDiego LaJolla,CA,USA FrontiersinProbabilityandtheStatisticalSciences ISBN978-3-319-21346-0 ISBN978-3-319-21347-7 (eBook) DOI10.1007/978-3-319-21347-7 LibraryofCongressControlNumber:2015948372 SpringerChamHeidelbergNewYorkDordrechtLondon ©TheAuthor2015 Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation, broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology nowknownorhereafterdeveloped. Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant protectivelawsandregulationsandthereforefreeforgeneraluse. Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinformationinthisbook arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor theeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinorforany errorsoromissionsthatmayhavebeenmade. Printedonacid-freepaper SpringerInternational PublishingAGSwitzerlandispartofSpringerScience+Business Media(www. springer.com) Fu¨rdiezwei ViolinenunddieViolades Model-FreienQuartetts Preface Predictionhasbeenoneoftheearliestformsofstatisticalinference.Theemphasis onparametricestimationandtestingseemsto onlyhaveoccurredabout100years ago;seeGeisser(1993)forahistoricaloverview.Indeed,parametricmodelsserved as a cornerstone for the foundation of Statistical Science in the beginning of the twentiethcenturybyR.A.Fisher,K.Pearson,J.Neyman,E.S.Pearson,W.S.Gosset (alsoknownas“Student”),etc.;theirseminaldevelopmentsresultedintoacomplete theoryofstatisticsthatcouldbepracticallyimplementedusingthetechnologyofthe time,i.e.,penandpaper(andslide-rule!). While some models are inescapable, e.g., modeling a polling dataset as a se- quence of independentBernoullirandom variables, others appear contrived,often invoked for the sole reason to make the mathematics work. As a prime example, theubiquitous—andtypicallyunjustified—assumptionofGaussiandatapermeates statisticstextbookstotheday.Modelcriticismanddiagnosticsweredevelopedasa practicalwayout;seeBox(1976)foranaccountofthemodel-buildingprocessby oneofthepioneersofappliedstatistics. With the advent of widely accessible powerful computing in the late 1970s, computer-intensivemethodssuchasresamplingandcross-validationcreatedarev- olutioninmodernstatistics. Usingcomputers,statisticiansbecameabletoanalyze big datasets for the first time, paving the way towards the “big data” era of the twenty-first century. But perhaps more importantwas the realization that the way wedotheanalysiscould/shouldbechangedaswell,aspractitionersweregradually freedfromthe limitationsofparametricmodels.Forinstance,thegreatsuccessof Efron’s(1979)bootstrapwasinprovidingacompletetheoryforstatisticalinference underanonparametricsettingmuchlikeMaximumLikelihoodEstimationhaddone halfacenturyearlierundertherestrictiveparametricsetup. Nevertheless,thereisafurtherstep onemaytake,i.e.,goingbeyondevennon- parametricmodels,andthisisthesubjectofthemonographathand.Toexplainthis, letusmomentarilyfocusonregression,i.e.,datathatarepairs:(Y ,X ),(Y ,X ),..., 1 1 2 2 (Y ,X ),whereY isthemeasuredresponseassociatedwitharegressorvalueofX. n n i i Thereareseveralwaystomodelsuchadataset;threemainonesarelistedbelow. vii viii Preface Theyallpertaintothestandard,homoscedasticadditivemodel: Y =μ(X)+ε (1) i i i wheretherandomvariablesεareassumedtobeindependent,identicallydistributed i (i.i.d.)fromadistributionF(·)withmeanzero. • Parametricmodel: Both μ(·)andF(·)belongtoparametricfamiliesoffunc- tions,e.g.,μ(x)=β +βxandF(·)isN(0,σ2). 0 1 • Semiparametricmodel: μ(·)belongstoaparametricfamily,whereasF(·)does not;instead,itmaybeassumedthatF(·)belongstoasmoothnessclass,etc. • Nonparametric model: Neither μ(·) nor F(·) can be assumed to belong to parametricfamiliesoffunctions. Despite the nonparametric aspect of it, even the last option constitutes a model, and is thus rather restrictive. To see why, note that Eq.(1) with i.i.d. errors is not satisfied inmanycases ofinterestevenafterallowingforheteroscedasticityofthe errors. For example, consider the modelY =G(X,ε), where the ε are i.i.d., and i i i i G(·,·)isanonlinear/non-additivefunctionoftwovariables.Itisforthisreason,i.e., to render the data amenable to an additive model such as (1), that a multitude of transformationsin regressionhavebeen proposedand studiedoverthe years,e.g., Box-Cox,ACE,AVAS,etc.;seeLintonetal.(1997)forareview. Nevertheless,itispossibletoshunEq.(1)altogetherandstillconductinference aboutaquantityofinterestsuchastheconditionalexpectationfunctionE(Y|X=x). Incontrasttononparametricmodel(1),thefollowingmodel-freeassumptioncanbe made: • Model-freeregression: – Randomdesign.Thepairs(Y ,X ),(Y ,X ),...,(Y ,X )arei.i.d. 1 1 2 2 n n – Deterministicdesign.ThevariablesX ,...,X aredeterministic,andtheran- 1 n dom variablesY ,...,Y are independentwith common conditionaldistribu- 1 n tion,i.e.,P{Y ≤y|X =x}=D (y)notdependingon j. j j x Inferenceforfeatures,i.e.,functionals,ofthecommonconditionaldistributionD (·) x is still possible under some regularity conditions, e.g., smoothness. Arguably,the mostimportantsuchfeatureistheconditionalmeanE(Y|X=x)thatcanbedenoted μ(x).Whileμ(x)iscrucialinthemodel(1)asthefunctionexplainingY onthebasis ofX =x,ithasakeyfunctioninmodel-freepredictionaswell: μ(x )isthemean f squared error (MSE) optimal predictor of a future response Y associated with a f regressorvaluex . f Aswillbeshowninthesequel,itispossibletoaccomplishthegoalofpointand intervalpredictionofY undertheabovemodel-freesetup;thisisachievedviathe f Model-freePredictionPrincipledescribedin PartI of thebook.In so doing,the solutiontointerestingestimationproblemsisobtainedasaby-product,e.g.,infer- ence on features of D (·); the prime example again is μ(x). Hence, a Model-free x approach to frequentist statistical inference is possible, including prediction and confidenceintervals. Preface ix Innonparametricstatistics,itiscommontotrytodevelopsomeasymptoticthe- oryfornewmethodsdeveloped.Inadditiontoofferingjustificationfortheaccuracy of these methods,asymptoticsoftenprovideinsightson practicalimplementation, e.g., on the optimal choice of smoothing bandwidth, etc. All of the methods dis- cussed/employedin the proposed Model-free approach to inference will be based on estimators that have favorable large-sampleproperties—such as consistency— under regularity conditions. Furthermore, asymptotic information on bandwidth rates,MSE decayrates, etc.will begivenwheneveravailablein the formof Facts orClaimstogetherwithsuggestionsontheirproofand/orreferences.However,for- maltheoremsandproofsweredeemedbeyondthescopeofthismonographinorder tobetterfocusonthemethodology,aswellaskeepthebook’slength(andtimeof completion)undercontrol.Perhapsmoreimportantly,notethatitisstillunclearhow toproperlyjudgethequalityofpredictionintervalsinanasymptoticsetting;some preliminaryideasonthisissuearegiveninSects.3.6.2and7.2.3,andtheRejoinder ofPolitis(2013). Interestingly,theemphasisonpredictionseemstobecomingbackfull-circlein thetwenty-firstcenturywiththerecentboominmachinelearninganddatamining; see, e.g., the highlyinfluentialbookonstatistical learningby Hastie etal. (2009), and the recent monograph on predictive modeling by Kuhn and Johnson (2013). TheModel-freepredictionmethodspresentedhereareofaverydifferentnaturebut share some similarities, e.g., in employing cross-validation and sample re-use for fine-tuning and optimization, and may thus complement well the popular model- basedapproachestopredictionandclassification.Furthermore,ideasfromstatisti- callearningandmodelselectioncouldeventuallybeincorporatedintheModel-free frameworkas well, e.g., selecting a subset of regressors;this is the subjectof on- goingwork.Notably,themethodspresentedinthismonographareverycomputer- intensive; relevant R functions and software are given at: http://www.math. ucsd.edu/˜politis/DPsoftware.html. I would like to thank my colleagues in the Departments of Mathematics and Economics of UCSD for their support, and my Ph.D. students for bearing with someofthematerial.Ihavebenefitedimmenselyfromsuggestionsanddiscussions withcolleaguesfromallovertheworld;averypartiallistincludes:IanAbramson, EryArias-Castro,BrendanBeare,PatriceBertail,RicardoCao,AnirbanDasGupta, RichardDavis,BradEfron,PeterHall,XumingHe,NancyHeckman,Go¨ranKauer- mann,ClaudiaKlu¨ppelberg,PiotrKokoszka,Jens-PeterKreiss,MicheleLaRocca, Jacek Leskow, Tim McMurry, George Michailidis, Stathis Paparoditis, Mohsen Pourahmadi, Jeff Racine, Joe Romano, Dimitrios Thomakos, Florin Vaida, Slava Vasiliev,PhilippeVieu,andMichaelWolf.Furtheracknowledgementsaregivenat theendofseveralchapters. In closing, I would like to thank the Division of Mathematical Sciences of the NationalScienceFoundationfortheircontinuingsupportwithmultiplegrants,the most recent ones being DMS-10-07513and DMS 13-08319,and the John Simon GuggenheimMemorialFoundationfora2011–2012fellowshipthathelpedmeget started on this monograph. I would also like to thank Marc Strauss and Hannah
The list of books you might like
