Table Of Content

Online Nonparametric Regression with General Loss Functions AlexanderRakhlin KarthikSridharan UniversityofPennsylvania CornellUniversity January28,2015 5 1 0 Abstract 2 This paper establishes minimax rates for onlineregressionwith arbitrary classes of functionsandgeneral n losses.1 Weshowthatbelowacertainthresholdforthecomplexityofthefunctionclass,theminimaxratesde- a pendonboththecurvatureofthelossfunctionandthesequentialcomplexitiesoftheclass.Abovethisthreshold, J thecurvatureofthelossdoesnotaffecttherates.Furthermore,forthecaseofsquareloss,ourresultspointtothe 6 interestingphenomenon:wheneversequentialandi.i.d. empiricalentropiesmatch,theratesforstatisticaland 2 onlinelearningarethesame. ] Inadditiontothestudyofminimaxregret,wederiveagenericforecasterthatenjoystheestablishedoptimal L rates. Wealsoprovidearecipefordesigningonlinepredictionalgorithmsthatcanbecomputationallyefficient M forcertainproblems. Weillustratethetechniquesbyderivingexistingandnewforecastersforthecaseoffinite expertsandforonlinelinearregression. . t a t s 1 Introduction [ 1 Westudytheproblemofpredictingareal-valuedsequencey1,...,yn inanon-linemanner.Attimet 1,...,n,the = v forecasterreceivessideinformationintheformofanelementx ofanabstractsetX. Theforecasterthenmakes t 8 apredictiony onthebasisofthecurrentobservationx andthedata{(x ,y )}t 1encounteredthusfar,andthen 9 observestheretsponsey . t i i i=−1 5 t 6 Suchaprboblemofsequence predictionisstudied intheliterature undertwodistinctsettings: probabilistic 0 anddeterministic[18]. Intheformersetting,whichfallswithinthepurviewoftimeseriesanalysis,onepositsa . parametricformforthedata-generatingmechanismandestimatesthemodelparametersbasedonpastinstances 1 0 andinputinformationinordertomakethenextprediction. Incontrast,inthedeterministicsettingoneassumes 5 nosuchprobabilisticmechanism. Instead,thegoalisphrasedasthatofpredictingaswellasthebestforecaster 1 fromabenchmarksetofstrategies. Thislattersetting—oftentermedpredictionofindividualsequences,oronline : v learning—isthefocusofthepresentpaper. Xi Welettheoutcome yt andthepredictionyt takevaluesinY ⊆RandY ⊆R,respectively. Formally,adeter- ministicpredictionstrategyisamapping(X Y)t 1 X Y. Weletthelossfunction(y ,y ) ℓ(y ,y )score − t t t t ar thequalityofthepredictiononasingleround×.b × → c 7→ Assumethatthetimehorizonn Z ,isknowntotheforeccaster. Theoverallqualityobftheforecabsteristhen evaluated againstthebenchmarkse∈tof+predictors, denoted asaclassF offunctions X Y. Thecumulative → regretoftheforecasteronthesequence(x ,y ),...,(x ,y )isdefinedas 1 1 n n c n n ℓ(y ,y ) inf ℓ(f(x ),y ). (1) t t t t t 1 −f Ft 1 X= ∈ X= b Theforecasteraimstokeepthedifferencein(1)smallforallsequences(x ,y ),...,(x ,y ). 1 1 n n ThecomparisonclassFencodesthepriorbeliefaboutthefamilyofpredictorsoneexpectstoperformwell.Ifa forecastingstrategyguaranteessmallregretforallsequences,andifF isagoodmodelforthesequencesobserved inreality,thentheforecastingstrategywillalsoperformwellintermsofitscumulativeerror. Infact,wecantake 1Thispaperbuildsuponthestudyofonlineregressionwithsquareloss,presentedbytheauthorsattheCOLT2014conference. 1 F tobeaclassofsolutions (thatis,forecastingstrategies)toasetofprobabilisticsourcesonewouldobtainby positingagenerativemodelofdata.Bydoingso,wearemodelingsolutionstothepredictionproblemratherthan modelingthedata-generatingmechanism. Wereferto[18,21]forfurtherdiscussionsonthis“duality”between theprobabilisticanddeterministicapproaches. To ensure that F captures the phenomenon of interest, we would like F to be large. However, increasing the“size”ofF likelyleadstolargerregret,asthecomparisontermin(1)becomessmaller. Ontheotherhand, decreasingthe“size”ofF makestheregretminimizationtaskeasier,yetthepredictionmethodislesslikelytobe successfulinpractice.Thisdichotomyisananalogueofthebias-variancetradeoffcommonlystudiedinstatistics. Acontributionofthispaperisananalysisofthegrowthofregret(withn)intermsofvariousnotionsofcomplexity ofF.Thetaskwasalreadyaccomplishedin[24]forthecaseofabsolutelossℓ(a,b) a b .Inthepresentpaper =| − | weobtainoptimalguaranteesforconvexLipschitzlossesunderverygeneralassumptions. Togivethereaderasenseoftheresultsofthispaper,westatethefollowinginformalcorollary.Letcomplexity ofF bemeasuredviasequentialentropy atscaleβ,tobedefinedbelow. (Forthereaderfamiliarwithcovering numbers,thisisasequentialanalogue—introducedin[24]—oftheclassicalKoltchinskii-Pollardentropy). Corollary1(Informal). SupposesequentialentropyatscaleβbehavesasO(β p),p 0.Thenoptimalregret − > • forpredictionwithabsolutelossgrowsasn1/2ifp (0,2),andasn1 1/pforp 2; − ∈ > • forpredictionwithsquarelossgrowsasn1 2/(2 p)ifp (0,2),andasn1 1/p forp 2. − + − ∈ > Moreover,theserateshavematching,sometimesmoduloalogarithmicfactor,lowerbounds. Thefirstpartofthiscorollaryisestablishedin[24].Thesecondpartrequiresnewtechniquesthattakeadvan- tageofthecurvatureofthelossfunction. Inanattempttoenticethereader,letusdiscusstwoconclusionsthatcanbedrawnfromCorollary1. First, the rates of convergence match optimal rates for excess square loss in the realm of distribution-free Statistical LearningTheorywithi.i.d.data,undertheassumptiononthebehaviorofempiricalcoveringnumbers[27].Hence, intheabsenceofagapbetweenclassicalandsequentialcomplexities(introducedlater)theregressionproblems in the two seemingly different frameworksenjoy the same rates of convergence. A deeper understanding of this phenomenonisofagreatinterest. The second conclusion concerns the same optimal rate n 1/p for both square and absolute loss for “rich” − classes(p 2). Informally,strongconvexityofthelossdoesnotaffecttherateofconvergenceforsuchmassive > classes.Ageometricexplanationofthisinterestingphenomenonrequiresfurtherinvestigation. We finish this introduction with a note about the generality of the setting proposed so far. Suppose X Yt,thespaceofallhistoriesofY-valuedoutcomes. Denotingx (y ,...,y ),yt 1,wemayvieweac=h t n t 1 t 1 − ∪f ≤F itselfasastrategythatmapshistory yt−1 toaprediction. Ensuri=ngthat xt i−snotarbitrarybutconsistent ∈ withhistoryonlymakesthetaskofregretminimizationeasier;theanalysisofthispaperforthiscasefollowsalong thesamelines,butweomittheextraoverheadofrestrictionsonx ’sandinsteadreferthereaderto[14,21]. t The paper is organizedas follows. Section 2 introduces the notation and then presents a brief overview of sequential complexities. Upper and lower bounds on minimax regretare established in Sections 3 and 4. We calculateminimaxratesforvariousexamplesinSection5.Wethenturntothequestionofdevelopingalgorithms inSection6.WefirstshowthatanalgorithmbasedontheRademacherrelaxationisadmissible(see[19])andyields theratesderivedinanon-constructivemannerinthefirstpartofthepaper. Weshowthatfurtherrelaxationsin finitedimensionalspaceleadtothefamousVovk-Azoury-Warmuthforecaster.Wealsoderiveapredictionmethod forfiniteclassF. 2 Preliminaries 2.1 AssumptionsandDefinitions WeassumethatthesetofoutcomesY isaboundedset,arestrictionthatcanberemovedbystandardtruncation arguments(seee.g. [12]). LetX besome setofcovariates, andletF beaclassoffunctions X Y forsome → c 2 Y R.Recalltheprotocoloftheonlinepredictionproblem:Oneachroundt {1,...,n},x X isrevealedtothe t ⊆ ∈ ∈ learnerwhosubsequentlymakesapredictiony Y.Theresponsey Y isrevealedafterthepredictionismade. t t ∈ ∈ c Thelossfunctionℓ(,y)isassumedtobeconvex. Let∂ ℓ(a,y)denoteanyelementofthesubdifferentialset a · (withrespecttofirstargument),andassumethbat c sup ∂ ℓ(a,y) G . a | |≤ <∞ a Y,y Y ∈ ∈ Weassumethatforanydistributionof y suppcortedonY,thereisaminimizerofexpectedlossthatisfiniteand belongstoY: Y argminEℓ(y,y) . c ∩ y R 6=; ∈ Givenay Y,theerrorofalinearexpansioncatatoapproximbatefunctionvalueatbisdenotedby ∈ b ∆ay,b , ℓ(b,y)−[ℓ(a,y)+∂aℓ(a,y)·(b−a)]. Let∆:(Y Y) R beafunctiondefinedpointwiseas 0 − → ≥ c c ∆(x) inf ∆y , (2) = a,b a,b Y,y Y s.t.b a x ∈ ∈ − = a lower bound onthe residualfor any twovalues secparated by x. For instance, aneasy calculation showsthat ∆(x) x2forℓ(y,y) (y y)2. = = − 2.2 MinimabxFormbulation Unlikemostpreviousapproachestothestudyofonlineregression,wedonotstartfromanalgorithm,butinstead workdirectlywithminimaxregret.Wewillbeabletoextracta(notnecessarilyefficient)algorithmafterobtaining upperboundsontheminimaxvalue.Letusintroducethenotationthatmakestheminimaxregretdefinitionmore n concise. Weuse todenoteaninterleavedapplicationoftheoperators, repeatedover t 1...n rounds. Withthisnotation⟪,·t·h·⟫etm=1inimaxregretoftheonlineregressionproblemdescribedearliercanbew=rittenas n n n V supinfsup ℓ(y ,y ) inf ℓ(f(x ),y ) (3) n t t t t =⟪ xt yt yt ⟫t=1(tX=1 −f∈FtX=1 ) b whereeach x rangesoverX, y rangebsoverY, and y rangesoverY. Anupper boundonV guaranteesthe t t t n existence ofanalgorithm(thatis, awaytochoose y ’s)withatmostthatmuchregretagainstanysequence. A t lowerboundonVn,inturn,guabranteestheexisctenceofasequenceonwhichnomethodcanperformbetterthan thegivenlowerbound. b 2.3 SequentialComplexities Oneofthekeytoolsinthestudyofestimatorsbasedoni.i.d. dataisthesymmetrizationtechnique[13]. Byin- troducingRademacherrandomvariables,onecanstudythesupremumofanempiricalprocessconditionallyon thedata. Conditioning facilitatestheintroductionofsample-basedcomplexities ofafunctionclass, suchasan empiricalcoveringnumber. Foraclassofboundedfunctions,thecoveringnumberwithrespecttotheempirical metricisnecessarilyfiniteandleadstoacorrectcontroloftheempiricalprocessevenifdiscretizationofthefunc- tionclassinadata-independentmannerisimpossible.Wewillreturntothispointwhencomparingourapproach withdiscretization-basedmethods. Intheonlinepredictionscenario,symmetrizationismoresubtleandinvolvesthenotionofabinarytree.The binarytreeis,insomesense,thesmallestentitythatcapturesthesequentialnatureoftheproblem.Moreprecisely, aZ-valuedtreezofdepthnisacompleterootedbinarytreewithnodeslabeledbyelementsofasetZ. Equiva- lently,wethinkofzasnlabelingfunctions,wherez isaconstantlabelfortheroot,z ( 1),z ( 1) Z arethelabels 1 2 2 − + ∈ 3 fortheleftandrightchildrenoftheroot,andsoforth. Hence,forǫ (ǫ ,...,ǫ ) { 1}n,z (ǫ) z (ǫ ,...,ǫ ) Z 1 n t t 1 t 1 isthelabelofthenodeonthet-thlevelofthetreeobtainedbyfollo=wingthepat∈hǫ±.Forafunc=tiong:Z −R,g∈(z) → isanR-valuedtreewithlabelingfunctionsg z forlevelt (or,inplainwords,evaluationofg onz). t ◦ Wenowdefinetwotree-basedcomplexitynotionsofaclassoffunctions. Definition1([24]). SequentialRademachercomplexityofaclassF RX onagivenX-valuedtreexofdepthn, ⊆ aswellasitssupremum,aredefinedas n R (F;x),Esup ǫ f(x (ǫ)) , R (F),supR (F;x) (4) n t t n n f F·t 1 ¸ x ∈ X= wheretheexpectationisoverasequenceofindependentRademacherrandomvariablesǫ (ǫ ,...,ǫ ). 1 n = Onemaythinkofthefunctionsx ,...,x asapredictableprocesswithrespecttothedyadicfiltration{σ(ǫ ,...,ǫ )} . 1 n 1 t t 1 Thefollowingnotionofaβ-coverquantifiescomplexityoftheclassF evaluatedonthepredictableprocess. ≥ Definition2([24]). AsetV ofR-valuedtreesofdepthnformsaβ-cover(withrespecttotheℓ norm)ofafunction q classF RX onagivenX-valuedtreexofdepthnif ⊆ 1 n f F, ǫ { 1}n, v V s.t. f(x (ǫ)) v (ǫ)q βq. t t ∀ ∈ ∀ ∈ ± ∃ ∈ n | − | ≤ t 1 X= Aβ-coverintheℓ senserequiresthat f(x (ǫ)) v (ǫ) βforallt [n].Thesizeofthesmallestβ-coverisdenoted t t byN (β,F,x),an∞dN (β,F,n),sup|N (β,F−,x). |≤ ∈ q q x q WewillrefertologN (β,F,n)assequentialentropyofF.Inparticular,wewillstudythebehaviorofV when q n sequentialentropygrowspolynomially2asthescaleβdecreases: logN2(β,F,n) β−p, p 0. (5) ∼ > Wealsoconsidertheparametric“p 0”casewhensequentialcoveringitselfbehavesas = N2(β,F,n) β−d (6) ∼ (e.g. linearregressioninaboundedsetinRd). Weremarkthattheℓ coverisnecessarilyn-dependent, sothe ∞ formsweassumefornonparametricandparametriccases,respectively,are logN (β,F,n) β−plog(n/β) or N (β,F,n) (n/β)d (7) ∞ ∼ ∞ ∼ 3 UpperBounds Thefollowingtheoremfrom[24]showstheimportanceofsequentialRademachercomplexityforpredictionwith absoluteloss. Theorem2([24]). LetY [ 1,1],F [ 1,1]X,andℓ(y,y) y y .Itthenholdsthat = − = − =| − | R (F) V 2R (F). n ≤ bn≤ bn Furthermore,anupperboundof2GR (F)holdsforanyG-Lipschitzloss. Weobserve,however,thatassoon n asF containstwodistinctfunctions,sequentialRadmeachercomplexityofF scalesasΩ(n1/2). Yet,itisknown thatminimaxregretforpredictionwithsquarelossgrowsslowerthanthisrate.Therefore,thedirectanalysisbased onsequentialRademachercomplexity(andacontractionlemma)giveslooseupperboundsonminimaxregret. ThekeycontributionofthispaperisanintroductionofanoffsetRademachercomplexitythatcapturesthecorrect behavior. Inthenextlemma,weshowthatminimaxvalueofthesequentialpredictionproblemwithanyconvexLipschitz lossfunctioncanbecontrolledviaoffsetsequentialRademachercomplexity. Asbefore,letǫ (ǫ ,...,ǫ )where 1 n = eachǫ isanindependentRademacherrandomvariable. i 2Itisstraightforwardtoallowconstantsinthisdefinition,andweleavethesedetailsoutforthesakeofsimplicity. 4 Lemma3. UndertheassumptionsanddefinitionsinSection2.1,theminimaxrateisboundedby n V supEsup 2Gǫ (f(x (ǫ)) µ (ǫ)) ∆ f(x (ǫ)) µ (ǫ) (8) n≤ x,µ f∈F·tX=1 t t − t − ³ t − t ´¸ wherexandµrangeoverallX-valuedandY-valuedtreesofdepthn,respectively. Theright-handsideof(8)willbetermedoffsetRademachercomplexity ofafunction classF RX withre- c ⊆ specttoaconvexevenoffsetfunction∆:R R andameanR-valuedtreeµ. If∆ 0,werecoverthenotionof 0 sequentialRademachercomplexitysinceE[→ǫ µ≥(ǫ)] 0. ≡ t t = A matching lower bound on the minimax value will be presented in Section 4, and the tworesults warrant a further study of offset Rademacher complexity. To this end, a natural next question is whether the chaining techniquecanbeemployedtocontrolthesupremumofthismodifiedstochasticprocess.Asapointofcomparison, wefirstrecallthatsequentialRademachercomplexityofaclassG of[ 1,1]-valuedfunctionsonZ canbeupper − boundedviatheDudleyintegral-typebound n 1 Esup ǫ g(z (ǫ)) inf 4ρn 12pn logN (δ,G,z)dδ , (9) t t 2 g∈G·tX=1 ¸≤ρ∈(0,1]½ + Zρ q ¾ foranyZ-valuedtreezofdepthn,asshownin[26].WeaimtoobtaintighterupperboundsontheoffsetRademacher bytakingadvantageofthenegativeoffsetterm. To initiate the study of offset Rademacher complexity with functions ∆ other than quadratic, we recall the notionofaconvexconjugate. Definition3. Foraconvexfunction ψ:D RwithdomainD R, theconvexconjugate ψ∗ :R R { }is → ⊆ → ∪ +∞ definedas ψ∗(a) sup ad ψ(d) . =d D − ∈ © ª Thechainingtechniqueforcontrollingasupremumofastochasticprocessrequiresastatementaboutthebe- havioroftheprocessoverafinitecollection.ThenextlemmaprovidessuchastatementfortheoffsetRademacher process. Lemma4. Let∆beaconvex,nonnegative,evenfunctiononRandletΓ denotetheconvexconjugateofthefunction ∗ x ∆(px ).AssumeΓ isnondecreasing.ForanyfinitesetW ofR-valuedtreesofdepthnandanyconstantC 0, ∗ 7→ | | > n 1 Emax 2Cǫtwt(ǫ) ∆(wt(ǫ)) inf log W nΓ∗ 2C2λ . (10) w∈W½tX=1 − ¾≤λ>0½λ | |+ ¡ ¢¾ Further,forany[ G,G]-valuedtreeη, − n n Emax ǫ η (ǫ)w (ǫ) G 2log W max w (ǫ)2. (11) w∈W·tX=1 t t t ¸≤ s | |·w∈W,ǫ1:ntX=1 n Asanexample,if∆(x) x2,aneasycalculationshowsthatΓ (1) 0andΓ (y) foranyy 1.Hence,the ∗ ∗ = = =+∞ 6= infimumin(10)isachievedatλ 1/(2C2),andtheupperboundbecomes2C2log W . = | | Wecannowemploythechainingtechniquetoextendthecontrolofthestochasticprocessbeyondthefinite collection. Lemma5. Let∆andΓ beasinLemma4.ForanyZ-valuedtreezofdepthnandaclassG offunctionsZ Rand ∗ → anyconstantC 0, > n γ Esup 2Cǫ g(z (ǫ)) ∆ g(z (ǫ)) inf C inf 4ρn 12pn logN (δ,G,z)dδ t t t g∈G·tX=1 − ¡ ¢¸≤γ>0( ρ∈(0,γ)½ + Zρ q ∞ ¾ 1 +λinf0½λlogN∞ γ2,G,z +nΓ∗ 2C2λ ¾¾. > ¡ ¢ ¡ ¢ 5 Remark1. Forthecaseof∆(x) x2,itispossibletoprovetheupperboundofLemma5intermsofℓ sequential 2 = coveringnumbersratherthanℓ (see[22]). ∞ Lemma5,togetherwithLemma3,yieldupperboundsonminimaxregretunderassumptionsonthegrowthof sequentialentropy.Beforedetailingtherates,wepresentlowerboundsontheminimaxvalueintermsoftheoffset Rademachercomplexityandcombinatorialdimensions. 4 LowerBounds Thefunction∆,arisingfromuniform(orstrong)convexityofthelossfunction,enterstheupperboundsonmini- maxregret.Forprovinglowerbounds,weconsiderthedualproperty,thatof(restricted)smoothness.Tothisend, letS Y beasubsetsatisfyingthefollowingcondition: ⊆ 1 c s S, y (s),y (s) Y s.t. s argmin ℓ(y,y (s)) ℓ(y,y (s)) . (12) 1 2 1 2 ∀ ∈ ∃ ∈ ∈ 2 + y Y ∈ ¡ ¢ b b ForanysuchsubsetS,let∆ :(Y S) R bedefinedas b c S 0 − → ≥ c ∆ (x) sup max ∆y1(s),∆y2(s) . (13) S = s,b s,b s∈S,b∈Y s.t.b−s=x n o Wewrite∆ forthesingletonsetS {κ}. c κ = Thelowerboundsinthissectionwillbeconstructedfromsymmetricdistributionssupportedontwocarefully chosenpoints.Crucially,wedonotrequireauniformnotionofsmoothness,butratheraconditiononthelossthat holdsforarestrictedsubsetSandatwo-pointdistribution. Asanexample,considersquarelossandY Y ( B,B). Foranys Y,wemaychoosethetwopointsas = = − ∈ s δ Y,forsmallenoughδ,withthedesiredproperty.ThenS Y and∆ (x) x2. S ± ∈ = = c c Lemma6. FixR 0.SupposeS satisfiescondition(12),andsupcposethatforanys S, > 6=; ∈ ∂ℓ(s,y (s)) R, ∂ℓ(s,y (s)) R. 1 2 =+ =− ThenforanyS-valuedtreeµofdepthn, n V supEsup ǫ R f(x (ǫ)) µ (ǫ) ∆ (f(x (ǫ)) µ (ǫ)) . (14) n≥ x f F·t 1 t t − t − µt(ǫ) t − t ¸ ∈ X= ¡ ¢ Thelowerboundin(14)isanoffsetRademachercomplexity thatmatchestheupper boundofLemma 3up toconstants,aslongasfunctions∆and∆exhibitthesamebehavior. Inparticular,theupperandlowerbounds matchuptoaconstantforthecaseofsquareloss. Ournextstepistoquantifythelowerboundintermsofnaccordingto“size”ofF.Incontrasttothemorecom- monstatisticalapproachesbasedoncoveringnumbersandFanoinequality,weturntoanotionofacombinatorial dimensionasthemaintool. Definition4. AnX-valuedtreeofdepthdissaidtobeβ-shatteredbyF ifthereexistsanR-valuedtreesofdepth d suchthat ǫ { 1}d, fǫ F s.t. ǫ (fǫ(x (ǫ)) s (ǫ)) β/2 t t t ∀ ∈ ± ∃ ∈ − ≥ forallt {1,...,d}.Thetreesiscalledawitness.Thelargestdforwhichthereexistsaβ-shatteredX-valuedtreeis ∈ calledthe(sequential)fat-shatteringdimension,denotedbyfat (F). β ThereaderwillnoticethattheupperboundofLemma5isintermsofsequentialentropiesratherthancombi- natorialdimensions.Thetwonotions,however,arecloselyrelated. 6 Theorem7([26]). LetF beaclassoffunctionsX [ 1,1].Foranyβ 0, → − > 2en fatβ(F) N (β,F,n) N (β,F,n) . 2 ≤ ∞ ≤ β µ ¶ Asaconsequenceoftheabovetheorem,iflogN (β,F,n) (c/β)p andβ 1/n,thenfat (F) (c /β)p/log(n) 2 β ′ ≥ ≥ ≥ wherec,c maydependontherangeoffunctionsinF. ′ The lower bounds will now be obtained assuming fat (F) β p behavior of the fat-shattering dimension, β − ≥ andthecorrespondingstatementsintermsofthesequentialentropygrowthwillinvolveextralogarithmicfactors, hiddenintheΩ()notation. · Lemma8. SupposethestatementofLemma6holdsforsomeR 0,andsuppose e > R ∆ (µ (ǫ) f(x (ǫ))) µ (ǫ) f(x (ǫ)) (15) µt(ǫ) t − t ≤ 2| t − t | forany f F andµ,xinthestatementofLemma6.Thenitholdsthatforanyβ 0andn fat (F), β ∈ > = V (R/2)nβ. n ≥ Inparticular,iffat (F) β p forp 0,wehave β − ≥ > 1 Vn (R/2)n−1/p . n ≥ Asanexample,considerthecaseofsquarelosswithY [ B,B]. ThenwemaytakeS {0},y B, y B, 1 2 = − = = =− andhenceR 2B.Weverifythat(15)holdsforY [ B/2,B/2]. = = − Lemma9. Suppose the statementof Lemma 6 holds for some R 0. For any class F and β 0, there exists a c ′ > > modifiedclassF suchthatforallβ β,fat (F ) fat (F) 2fat (F ) 4andforn fat (F), ′< β′ ′ ≤ β′ ≤ β′ ′ + > β 1V sup Rβ fatβ(F) ∆ β . n n≥  2 s 2n − κ 2  β,κ  µ ¶ ArmedwiththeupperboundsofSection3andthelowerboundsofSection4,wearereadytodetailspecific minimaxratesofconvergenceforvariousclassesofregressionfunctionsF andarangeoflossfunctionsℓ. 5 MinimaxRates CombiningLemma3andLemma5,wecandetailthebehaviorofminimaxregretunderanassumptionaboutthe growthrateofsequentialentropy. Theorem10. Letr 2,p 0andsupposethelossfunctionandthefunctionclassaresuchthat ≥ > ∆(t) Ktr, logN (β,F,n) β−plog(n/β). ≥ ∞ ≤ Thenforp (0,2), ∈ n1Vn≤min cr,p n−2(r−r1)+pG2(r−21r)+pK−2(r2−−1p)+p log(n), cFGlog1/2(n)n−1/2 . (16) ½ ¾ andforp 2 > 1 Vn cpGlog1/2(n)n−1/p (17) n ≤ Here,cF dependsonsupf∈F|f|∞.Atp=2,thebound(17)gainsanextralog(n)factor. 7 Wematchtheaboveupperboundswithlowerboundsundertheassumptiononthegrowthofthecombinato- rialdimension. Theorem11. Supposethe statementofLemma6holds forsomeR 0andκ S Y. Letr 2, p (0,2), and > ∈ ⊆ ≥ ∈ assume ∆κ β/2 Kβr, fatβ β−p. c ≤ ≥ Thenthereexistsafunctionclasssuchthatfor¡some¢constantcp,r 0, > n1Vn≥cp,rmin n−2(r−r1)+pR2(r−21r)+pK−2(r2−−1p)+p , Rn−1/2 . ½ ¾ forp (0,2).Furthermore,forp 2,foranyF withfatβ β−p, ∈ > ≥ 1 Vn (R/2)n−1/p n ≥ undertheassumption(15). ThelowerboundofTheorem11matches(uptopolylogarithmicinnfactors)theupperboundofTheorem10 in its dependence on n, the dependence onthe constant K, and in dependence onthe size ofthe gradientsG (respectively,R). Therestofthissectionisdevotedtothediscussionofthederivedupperandlowerboundsfor particularlossfunctionsorparticularclassesoffunctions. 5.1 Absoluteloss Weverifythatthegeneralstatementsrecoverthecorrectratesforthecaseofℓ(y,y) y y . Sincetheabsolute =| − | loss is not strongly convex, we take K 0 (and ∆ 0). Theorem 10 then yields the O(n 1/2) rate for p (0,2) − = ≡ ∈ and O(n−1/p) for p 2, up tologarithmic factors. These rates arematched, agbainupbtologarithmic factors, in > Theorem11.Ofcourse,theresultalreadyfollowsfromTheorem2. e Iteisalsoinstructivetocheckthecaseofr . Inthiscase,ifK isscaledproperlybytherangeoffunction →∞ values,thefunction∆approachesthezerofunction,indicatingabsenceofstrongconvexityoftheloss. Examin- ingthepower r inTheorem10, weseethatitapproaches1/2, matchingthediscussionofthepreceding 2(r 1) p paragraph. − + 5.2 Squareloss Thecaseofsquarelossℓ(y,y) (y y)2hasbeenstudiedin[22].InviewofRemark1,westatethecorollarybelow = − intermsofℓ coveringnumbers,thusremovingsomelogarithmictermsofTheorem10. 2 Corollary12. ForaclassFb withsbequentialentropygrowthlogN2(β,F,n) β−p, ≤ • Forp 2,theminimaxregret3isboundedas 1V Cn 1/p > n n≤ − • Forp (0,2),theminimaxregretisboundedas 1V Cn 2/(2 p) ∈ n n≤ − + • Fortheparametriccase(6), 1V Cdn 1log(n) n n≤ − • ForfinitesetF, 1V Cn 1log F n n≤ − | | Corollary13. TheupperboundsofCorollary12aretight4: • Forp 2,foranyclassF ofuniformlyboundedfunctionswithalowerboundofβ p onsequentialentropy − ≥ growth, 1V Ω(n 1/p) n n≥ − • Forp (0,2],foranyclassF ofuniformlyboundedfunctions,thereexistsaslightlymodifiedclassF withthe ∈ e ′ samesequentialentropygrowthsuchthat 1V Ω(n 2/(2 p)) n n≥ − + • ThereexistsaclassF withthecoveringnumberasin(6),suchthat 1V Ω(dn 1log(n)) e n n≥ − 34FTohrepΩ=()2n,on1tVatnio≤nCsulopgp(rne)sns−es1/l2o.garithmicfactors · e 8 5.3 q-lossforq (1,2) ∈ Consider the caseof ℓ(yˆ,y) y yˆq, for q (1,2), which interpolates between theabsolute value and square =| − | ∈ losses. Corollary14. SupposeY Y [ 1,1]andℓ(yˆ,y) y yˆq forq (1,2).AssumecomplexityofF asinTheorems = = − =| − | ∈ 10and11forsomep 0.Then > c n1Vn=Θ min q−1 −22+−pp n−2+2p,n−1/2 µ ½ ¾¶ ¡ ¢ 5.4 q-lossforq 2 ≥ Itiseasytocheckthatforq 2,ℓ(,y) y q isq-uniformlyconvex,andthus > · =|·− | ∆(t) Ctq ≥ Theupperboundof 1 q nVn≤Cn−2(q−1)+p thenfollowsfromTheorem10. 5.5 Logisticloss Thelossfunctionℓ(yˆ,y) log(1 exp{ yˆy})isstronglyconvexandsmoothifthesetsY,Y arebounded.Thiscan = + − beseenbycomputingthesecondderivativewithrespecttothefirstargument: c exp{yˆy} ℓ′′(yˆ,y) y2 = (1 exp{yˆy})2 + Weconcludethat n1Vn=Θ min n−2+2p,n−1/2 ³ n o´ Logisticlossisanexampleofafunctionwithethirdderivativeboundedbyamultipleofthesecondderivative. ControloftheremainderterminTaylorapproximationforsuchfunctionsisgivenin[5,Lemma1].Otherexamples ofstronglyconvexandsmoothlossesaretheexponentiallossandtruncatedquadraticloss.Theseenjoythesame minimaxrateasgivenabove. 5.6 Logarithmicloss Thetechniquedevelopedinthispaperisnotuniversal.Inparticular,itdoesnotyieldcorrectratesforrichclasses offunctionsundertheloss ℓ(y,y) log(y)1 y 1 log(1 y)1 y 0 =− = − − = fortheproblemofprobabilityassignmentandabina©ryalpªhabetY {0,©1}. ThªesuboptimalityofLemma3isdue b b =b totheexplodingLipschitzconstant.However,amodifiedapproachispossible,andwillbecarriedoutinaseparate paper. 5.7 Sparselinearpredictorsandsquareloss Wenowfocusonquadraticlossandinsteaddetailminimaxratesforspecificclassesoffunctions. Considerthe followingparametricclass. LetG {g ,...,g }beasetofM functionssuchthateachg :X [ 1,1]. DefineF 1 M i = 7→ − tobetheconvexcombinationofatmostsoutoftheseMfunctions.Thatis s s F α g :σ [M], j,α 0, α 1 =(j 1 j σj 1:s⊂ ∀ j ≥ j 1 j = ) X= X= 9 Forthisexamplenotethatthesequentialcoveringnumbercanbeeasilyupperbounded:wecanchoosesoutofM functionsin M waysandobservethatpointwisemetricentropyforconvexcombinationofsboundedfunctions s atscaleβisboundedasβ s.Weconcludethat ¡ ¢ − eM s N2(β,F,n) β−s ≤ s µ ¶ Fromthemaintheorem,forthecaseofsquareloss,theupperboundis slog(M/s) 1V O . n n≤ n µ ¶ Theextensiontootherlossfunctionsfollowsimmediatelyfromthegeneralstatements. 5.8 Besovspacesandsquareloss LetX beacompactsubsetofRd.LetF beaballinBesovspaceBs (X).Whens d/p,pointwisemetricentropy p,q > boundsatscaleβscalesasΩ(β d/s)[31,p. 20]. Ontheotherhand,whens d/p,andp 2,onecanshowthat − < > thespaceisap-uniformlyconvexBanachspace. From[26],itcanbeshownthatsequentialRademachercanbe upperboundedbyO(n1−1/p),yieldingaboundonminimaxrate. Thesetwocontrolstogethergivetheboundon theminimaxrate.ThegenericforecasterwithRademachercomplexityasrelaxation(seeSection6),enjoysthebest ofbothoftheserates.Morespecifically,wemayidentifythefollowingregimes: • Ifs≥d/2,theminimaxrateis n1Vn≤O n−2s2+sd . ³ ´ • Ifs d/2,theminimaxratedependsontheinteractionofpandd,s: < – ifp>ds,theminimaxrate n1Vn≤O n−ds , otherwise,therateis n1Vn≤O n−p1 ³ ´ ³ ´ 5.9 Remarks:Experts,Mixability,andDiscretization Theproblemofpredictionwithexpertadvicehasbeencentralintheonlinelearningliterature[9].Onecanphrase theexpertsprobleminoursettingbytakingafiniteclassF {f1,...,fN}offunctions.Itispossibletoensuresub- = linearregretbyfollowingthe“advice” fIt(xt)ofarandomlychosen“expert”It fromanappropriatedistribution overexperts. Therandomizedapproach,however,effectivelylinearizestheproblemanddoesnottakeadvantage ofthecurvatureoftheloss.Theprecisewayinwhichthelossentersthepicturehasbeeninvestigatedthoroughly byVovk[28](seealso[15]). Vovkdefinesamixabilitycurvethatparametrizesachievableregretofaformslightly differentthan(1).Specifically,Vovkallowsaconstantotherthan1infrontoftheinfimumintheregretdefinition. Suchregretboundsarecalled“inexactoracleinequalities”instatistics.Audibert[2]showsthatthemixabilitycon- ditiononthelossfunctionleadstoavariance-typeboundinhisgeneralPAC-basedformulation,yettheanalysis isrestrictedtothecaseoffiniteexperts.Whileitispossibletorepeattheanalysisinthepresentpaperwithacon- stantotherthan1infrontofthecomparator,thisgoesbeyondthescopeofthepaper.Importantly,ourtechniques gobeyondthefinitecaseandcangivecorrectregretboundsevenifdiscretizationtoafinitesetofexpertsyields vacuousbounds. Let us emphasize the abovepoint again by comparing the upper bound of Lemma 5 to the bound wemay obtainviaametricentropyapproach,asintheworkof[31].AssumethatF isacompactsubsetofC(X)equipped withsupremumnorm.Themetricentropy,denotedbyH(ǫ,F),isthelogarithmofthesmallestǫ-netwithrespect tothesupnormonX. Anaggregatingprocedureovertheelementsofthenetgivesanupper bound(omitting constantsandlogarithmicfactors) nǫ H(ǫ,F) (18) + onregret(1). Here,nǫistheamountwelosefromrestrictingtheattentiontotheǫ-net,andthesecondtermap- pearsfromaggregationoverafiniteset. Thebalance(18)failstocapturetheoptimalbehaviorforlargenonpara- p metricsetsoffunctions.Indeed,foranO(ǫ−p)behaviorofmetricentropy,VovkconcludestherateofO np+1 .For ³ ´ 10