Reproducing Kernel Banach Spaces with the l1 Norm II: Error Analysis for Regularized Least Square Regression

0.17 MB·English

by  Guohui Song

Reproducing Kernel Banach Spaces with the ℓ1 Norm II: Error
Analysis for Regularized Least Square Regression
∗
Guohui Song† and Haizhang Zhang‡
1
1
0
2
Abstract

n
a A typicalapproachin estimating the learning rate of a regularizedlearning scheme is to bound
J the approximationerrorby the sum ofthe sampling error,the hypothesis errorand the regulariza-

7 tionerror. Usingareproducingkernelspacethatsatisfiesthelinearrepresentertheorembringsthe
2 advantageof discarding the hypothesis errorfromthe sum automatically. Following this direction,
1
we illustrate how reproducing kernel Banach spaces with the ℓ norm can be applied to improve
] 1
L the learning rate estimate of ℓ -regularizationin machine learning.
M
Keywords: reproducingkernelBanachspaces,sparselearning,regularization,leastsquareregres-
. sion, learning rate, the representer theorem
t
a
t
s
[

1 Introduction
2
v
9 A class of reproducing kernel Banach spaces (RKBS) with the ℓ1 norm that satisfies the linear rep-
3
4 resenter theorem was recently constructed in [14]. The purpose of this note is to illustrate how the
4 obtainedspacescanbeappliedtoestimatethelearningrateoftheℓ1-regularizedleastsquareregression
.
1 in machine learning.
0
A general coefficient-based regularization of the least square regression has the form
1
1
m
v: min 1 Kx(x )c y 2+λφ(c), (1.1)
j j
i c∈Rm m | − |
X j=1
X
r
a where x := x : j N with N := 1,2,...,m is the sequence of sampling points from an input
j m m
{ ∈ } { }
space X, y Y R is the observed data on x , λ is a positive regularization parameter, φ is a
j j
∈ ⊆
nonnegative regularization function on the coefficient column vector c, and with a chosen function
K : X X R, Kx(x) is the 1 m row vector (K(x ,x) : j N ).
j m
× → × ∈
When K is a positive-definite reproducing kernel on X and
φ(c):= cTK[x]c, (1.2)
∗SupportedbyGuangdongProvincialGovernmentofChinathroughthe“ComputationalScienceInnovativeResearch
Team” program.
†School of Mathematical and Statistical Sciences, Arizona State University, Tempe, AZ 85287. E-mail address:
[e-mail].
‡School of Mathematics and Computational Science and Guangdong Province Key Laboratory of Computational
Science, Sun Yat-sen University,Guangzhou 510275, P.R. China. E-mail address: [e-mail].
1
where K[x] is the m m matrix defined by
×
(K[x]) := K(x ,x ), j,k N ,
j,k k j m
∈
it follows from the celebrated representer theorem [7] that (1.1) is the classical regularization network
and has been extensively studied in the literature [6, 9, 10, 13, 19]. Estimates for the learning rate
of the regularization network can be found, for example, in [4, 5, 12, 15, 23]. Learning rates for
(1.1) when φ(c) = m c p for 1 < p 2 and p = 2 were respectively obtained in [18] and [16].
j=1| j| ≤
The linear programming regularization where φ(c) is the ℓ1 norm c of c has recently attracted
P k k1
much attention. The increasing interest is mainly brought by the progress of the lasso in statistics
[17] and compressive sensing [2, 3] in which ℓ1-regularization is able to yield sparse representation
of the resulting minimizer, a desirable feature in model selection. Moreover, the ℓ1-regularization is
particularly robust to non-Gaussian additive noise such as impulsive noise [1, 8].
Without making use of a reproducing kernel space, the recent references [11, 20] established esti-
mates of the learning rate for the ℓ1-regularized least square regression
m
1
min Kx(x )c y 2+λ c . (1.3)
j j 1
c∈Rm m | − | k k
j=1
X
We attempt to show that improvement on the estimates could be made if an RKBS with the ℓ1 norm
is used. To explain how this could be done, we first introduce the popular approach [5] for learning
rate estimates in machine learning.
A fundamental assumption in machine learning is that the sample data z := (x ,y ): j N
j j m
{ ∈ }∈
X Y consistsofindependentandidentically distributedinstancesofarandomvariable(x,y) X Y
× ∈ ×
subject to an unknown probability measure ρ on X Y. The performance of a predictor f : X Y
× →
is hence measured by
(f) := f(x) y 2dρ.
E | − |
ZX×Y
The predictor that minimizes the above error is the regression function
f (x):= ydρ(y x), x X, (1.4)
ρ
| ∈
ZY
where ρ(y x) denotes the conditional probability measure of y with respect to x. In fact, we have for
|
every predictor f that
(f) = (f )+ f f 2 , (1.5)
E E ρ k − ρkL2ρX
p
where ρ is the marginal probability measure of ρ on X and for p [1,+ ), L denotes the Banach
X ∈ ∞ ρX
space of measurable functions f on X with respect to ρ such that
X
1/p
kfkLpρX := (cid:18)ZX |f(x)|pdρX(x)(cid:19) < +∞.
The formula (1.4), though attractive, is only of theoretical value as ρ is unknown. A practical way is
to find a minimizer cz,λ of (1.1) and hope that
x
fz,λ(x):= K (x)cz,λ, x X (1.6)
∈
will be competitive with fρ in the sense that the approximation error (fz,λ) (fρ) would be small.
E −E
To be more precise, for the learning scheme (1.1) to be useful in practice, this error should converge
to zero fast in probability as the number of sampling points increases.
2
Theapproach in[5] works by introducingintermediate functions between fz,λ andfρ that arefrom
a Banach space of functions on X with the properties that K(x, ) for all x X and for all
B · ∈ B ∈
pairwise distinct x X, j Nm and c Rm
j
∈ ∈ ∈
x
ψ( K ()c )= φ(c),
B
k · k
for some nonnegative function ψ. Here is the norm on . Let g be an arbitrary function from
B
k·k B
such a space and set for each function f :X R
B →
m
1
z(f):= (f(xj) yj)2.
E m −
j=1
X
The approximation error (fz,λ) (fρ) can then be decomposed into the sum of four quantities
E −E
(fz,λ) (fρ) = (z,λ,g)+ (z,λ,g)+ (λ,g) λψ( fz,λ B), (1.7)
E −E S P D − k k
where
(z,λ,g) := (fz,λ) z(fz,λ)+ z(g) (g),
S E −E E −E
(z,λ,g) := ( z(fz,λ)+λψ( fz,λ B)) ( z(g)+λψ( g B)),
P E k k − E k k
(λ,g) := (g) (f )+λψ( g ).
ρ B
D E −E k k
The above three quantities are called the sampling error, the hypothesis error and the regularization
error, respectively. The strategy is to choose and g carefully so that these three errors can be well
B
boundedfromabove. When isthereproducingkernelHilbertspaceofapositive-definitereproducing
B
kernel K on X and the regularizer φ is given by (1.2), we have ψ(t) = t2, t R and by the representer
∈
theorem and the definition of fz,λ in (1.6) that
fz,λ = argmf∈iBn Ez(f)+λkfk2B . (1.8)
(cid:0) (cid:1)
In this case, one immediately has that (z,λ,g) 0 and thus, by (1.7) that
P ≤
(fz,λ) (z,λ,g)+ (λ,g). (1.9)
E ≤ S D
For the ℓ1-regularization where φ(c)= c , the space chosen in [11, 20] does not satisfy the linear
1
k k B
representer theorem. Consequently, the hypothesis error needed to be dealt with there.
A class of RKBS with the ℓ1 norm that satisfies the linear representer theorem was recently
constructed in [14]. In Section 2, we shall follow a similar idea to construct a slightly larger RKBS
with the same desirable properties. By using the constructed space, we enjoy the same advantage
as that for the RKHS case of discarding the hypothesis error automatically. Moreover, the space
also leads to a better estimate of the regularization error than that in [20]. Combining these two
improvements and directly using the estimates of the sampling error established in [20] or [11], one
immediatelyhasasuperiorlearningrate. Asourfocusisontheadvantagesbroughtbytheconstructed
RKBS, we shall only improve the learning rate estimate of [20] in Section 3. Interested readers may
follow our strategy to engage the more sophisticated sampling error estimate given in [11] to improve
the learning rate therein.
2 RKBS by Borel Measures
In this section, we construct RKBS applicable to the error analysis of the ℓ1-regularized least square
regression. The constructed spaces are expected to have the ℓ1 norm and satisfy the linear representer
3
theorem. The approach is different from the one by semi-inner products in [21, 22] as an infinite-
dimensional ℓ1 space is neither reflexive nor strictly convex.
Suppose that the input space X is a locally convex topological space and denote by C (X) the
0
space of continuous functions f : X R such that for all ε > 0, the set x X : f(x) > ε is
→ { ∈ | | }
compact. We also impose the requirement that for all pairwise distinct x X, j N , m N, the
j m
∈ ∈ ∈
kernel matrix K[x] is nonsingular. With the maximum norm f := max f(x), the space
k kC0(X) x∈X| |
C (X) is a Banach space. Its dual space is isometrically isomorphic to the space (X) of all the
0
M
signed Borel measures on X with bounded total variation. In other words, for each continuous linear
functional T on C (X), there exists a unique measure µ (X) such that
0
∈ M
Tf
T(f)= f(x)dµ(x) and sup | | = µ , (2.1)
f k k
ZX f∈C0(X),f6=0 k kC0(X)
where µ denotes the total variation of µ.
k k
Let K be a real-valued function on X X such that K(,x) C (X) for all x X and
0
× · ∈ ∈
span K(,x) : x X = C (X). (2.2)
0
{ · ∈ }
With such a function, we introduce the following space
:= f := K(t, )dµ(t) : µ (X) (2.3)
µ
B · ∈ M
(cid:26) ZX (cid:27)
with the norm
f := µ . (2.4)
µ B
k k k k
Recall that a vector space V is called a pre-RKBS [14] on X if it is a Banach space consisting of
functions on X such that point evaluation functionals are continuous on V and such that for all
f V, f = 0 if and only if f vanishes everywhere on X.
V
∈ k k
Proposition 2.1 Suppose that K(,x) C (X) for all x X and (2.2) is satisfied. Then defined
0
· ∈ ∈ B
by (2.3) is a pre-RKBS on X.
Proof: We first show that the norm (2.4) is well-defined. Let µ,ν be two measures in (X) such that
M
f (x) = f (x) for all x X. Then we get that
µ ν
∈
K(t,x)d(µ ν)(t) = 0 for all x X.
− ∈
ZX
By the denseness condition (2.2), the above equation implies that µ ν = 0. Thus, the measure µ
−
associated with a function f is unique. This proves that (2.4) is well-defined and that f = 0
µ µ B
∈ B k k
if and only if f (x) = 0 for all x X. Another consequence is that is isometrically isomorphic to
µ
∈ B
(X) and is hence a Banach space. Finally, we observe for all x X and µ (X) that
0
M ∈ ∈M
f (x ) = K(t,x )dµ(t) K(,x ) µ = K(,x ) f .
| µ 0 | 0 ≤ k · 0 kC0(X)k k k · 0 kC0(X)k µkB
(cid:12)ZX (cid:12)
(cid:12) (cid:12)
(cid:12) (cid:12)
Therefore,pointevaluationsarecontinuouslinearfunctionalson . Weconcludethat isapre-RKBS
(cid:12) (cid:12)
B B
on X. The proof is complete. 2
4
Let the sampling points in x be pairwise distinct. By definition, Kx()c for all c Rm. The
· ∈ B ∈
denseness condition (2.2) implies that K(x , ), j N are linearly independent. As a result,
j m
· ∈
x
K ()c = c . (2.5)
B 1
k · k k k
It is in the above sense that is said to possess the ℓ1 norm.
B
We next turn to the crucial linear representer theorem in . We say that satisfies the lin-
B B
ear representer theorem if for all continuous nonnegative loss function Q and regularizer ψ with
lim ψ(t) = + , the regularized learning scheme
t→∞
∞
inf Q(f(x))+λψ( f )
B
f∈B k k
has a minimizer f of the form f = Kx()c for some c Rm. Here, f(x)= (f(x ) : j N )T.
0 0 j m
· ∈ ∈
The following lemma can be proved by arguments similar to those in [14].
Lemma 2.2 The space satisfies the linear representer theorem if and only if for all x of pairwise
B
distinct sampling points and y Rm, the minimal norm interpolation
∈
inf f : f , f(x)= y (2.6)
B
{k k ∈B }
has a minimizer f of the form f = Kx()c for some c Rm.
0 0
· ∈
A subspace of was constructed in [14] and conditions for it to satisfy the linear representer
B
theorem were studied. In order to make use of the results obtained there, we first introduce the
subspace. Denote by ℓ1(X) the subset of (X) of those Borel measures that are supported on a
M
countable subset of X. Thus, for each ν ℓ1(X), there exist some pairwise distinct points x X,
j
∈ ∈
j I where I is a countable index set, such that
∈
ν(A) = ν(x ) for every Borel subset A X.
j
⊆
xXj∈A
Denote by suppν the countable set of points where ν is nonzero. The space considered in [14] is
1
B
:= ν(x)K(x, ) :ν ℓ1(X)
1
B · ∈
( )
x∈suppν
X
with the norm inherited from that of .
B
Put for all x X, Kx(x) := (K(x,xj) : j Nm)T, which is an m 1 vector in Rm. One should
∈ x ∈ ×
not confuse Kx(x) with K (x). The latter is 1 m and might even not be the transpose of the former
×
as K is not required to be symmetric. The following result about is from [14].
1
B
Lemma 2.3 For all y Rm, the minimal norm interpolation
∈
inf f :f , f(x)= y (2.7)
{k kB1 ∈B1 }
has a minimizer f of the form f = Kx()c for some c Rm if and only if
0 0
· ∈
K[x]−1Kx(x) 1 1 for all x X. (2.8)
k k ≤ ∈
Moreover, under condition (2.8), there holds for all c Rm that
∈
kcTKx(·)kC0(X) = kcTK[x]k∞, (2.9)
where is the maximum norm on Rm.
∞
k·k
5
We are ready to present the main result of this section.
Theorem 2.4 The space satisfies the linear representer theorem if and only if (2.8) holds true.
B
Proof: Suppose that (2.8) holds true. By Lemma 2.2, to show that satisfies the linear representer
theorem, it suffices to show that f = Kx()K[x]−1y is a minimizer oBf (2.6). Clearly, f (x) = y. Let
0 0
·
f , µ (X), be an arbitrary function in that satisfies the interpolation condition f (x) = y. We
µ µ
∈ M B
then have for all c Rm that
∈
m m
cTKx(t)dµ(t) = cjK(t,xj)dµ(t) = cjfµ(xj) = cTy.
ZX ZX j=1 j=1
X X
It follows from (2.1) that for all c Rm
∈
|cTy| ≤ kcTKx(·)kC0(X)kµk.
This together with (2.9) implies that
cTy cTy aTK[x]−1y
µ sup | | = sup | | = sup | | = K[x]−1y .
k k ≥ c∈Rm,c6=0kcTKx(·)kC0(X) c∈Rm,c6=0 kcTK[x]k∞ a∈Rm,a=6 0 kak∞ k k1
Now, recall by (2.5) that f = K[x]−1y and by definition of that f = µ . These two
0 B 1 B µ B
k k k k k·k k k k k
facts combined with the above inequality imply that f f . Thus, f is indeed a minimizer
µ B 0 B 0
k k ≥ k k
of (2.6).
On the other hand, suppose that satisfies the linear representer theorem and we want to prove
B
(2.8). Let y Rm. By Lemma 2.2, the minimal norm interpolation (2.6) has a minimizer f of the
0
form f = Kx∈()c for some c Rm. Clearly, f is also a minimizer of (2.7) because f and
0 0 0 1
· ∈ ∈ B
f = f = inf f : f , f(x)= y inf f :f , f(x) = y .
k 0kB1 k 0kB {k kB ∈ B } ≤ {k kB1 ∈B1 }
By Lemma 2.3, (2.8) holds true. The proof is complete. 2
It will become clear in the next section that the above theorem makes a useful space for error
B
analysis of the ℓ1-regularized least square regression.
We present two examples of K that satisfy all the assumptions, especially (2.8), in this section:
– the exponential kernel
K(s,t) := e−|s−t|, s,t R,
∈
– the Brownian bridge kernel
K(s,t) := min s,t st, s,t (0,1).
{ }− ∈
That these two kernels satisfy (2.8) has been proved in [14]. It remains to verify the denseness
requirement (2.2). The exponential kernel is a particular case of the following result.
Proposition 2.5 IfφisLebesgueintegrable onRd that isnonzero almost everywhere thenthe function
K(s,t):= e−i(s−t)·ξφ(ξ)dξ, s,t Rd (2.10)
Rd ∈
Z
satisfies that K(,t) C (Rd) for all t Rd and the denseness condition (2.2). So does K(s,t) :=
0
· ∈ ∈
ψ(s t), s,t Rd where ψ is a nontrivial continuous function on Rd of compact support.
− ∈
6
Proof: That the function given by (2.10) belongs to C (Rd) for all t Rd follows from the Riemann-
0
∈
Lebesgue lemma. The denseness condition (2.2) for the two kernels can be proved by arguments
similar to those in [14]. 2
The Brownian bridge kernel is handled with a manner different from that in [14].
Proposition 2.6 The Brownian bridge kernel satisfies (2.2).
Proof: Clearly, for the Brownian bridge kernel, K(,t) is continuous for all t (0,1). Let ν be a Borel
· ∈
measure on X := (0,1) such that
K(s,t)dν(s)= 0 for all t (0,1). (2.11)
∈
ZX
Note that K has the representation
K(s,t)= Γ (z)Γ (z)dz, s,t (0,1),
s t
∈
ZX
where Γ := χ s with χ denoting the characteristic function of (0,s). Arguments similar to
s (0,s) (0,s)
−
those in [14] yield that there exists a constant C such that
s
dν(s) = C for all s (0,1).
∈
Z0
It follows that ν((s ,s )) = 0 for all 0 < s < s < 1. Consequently, ν is the zero Borel measure on
1 2 1 2
(0,1). Thus, the Brownian bridge kernel satisfies (2.2). 2
Finally, weremarkthatthefunctionK canberegardedasthereproducingkernelfor constructed
B
by (2.3). To see this, we introduce a bilinear form on C (X) by setting
0
B×
f ,g := g(x)dµ(x) for all µ (X) and g C (X).
µ 0
h i ∈M ∈
ZX
We observe by (2.1) that
f ,g µ g = f g
|h µ i| ≤ k kk kC0(X) k µkBk kC0(X)
and that for all x X,
∈
f (x) = f ,K(,x) , g(x) = K(x, ),g .
µ µ
h · i h · i
In the above senses, K is said to be the reproducing kernel for both and C (X).
0
B
3 Error Analysis of the ℓ1-Regularization
We apply the constructed space to estimate the learning rate of the ℓ1-regularized least square
B
regression (1.3) in this section. To this end, we first introduce some standard assumptions in the
literature imposed on the regression function f , the input space X and the function K.
ρ
Let X be compact metric space with the distance d and assume that ρ is a Borel probability
X
measure on X. In this note, we suppose that K is a positive-definite reproducing kernel on X with
the Lipschitz condition
K(x,t) K(x,t′) C (d(t,t′))α for some positive constants α,C and for all x,t,t′ X. (3.1)
α α
| − | ≤ ∈
7
Denote for all r > 0 by (X,r) the least number of open balls with radius r that cover X. Assume
N
that this covering number satisfies for some positive constants η,C that
η
C
η
(X,r) for all 0 < r 1. (3.2)
N ≤ rη ≤
The requirement on f is that it is contained in the range ran(Ls ) of Ls for some s > 0. Here, L
ρ K K K
is the compact positive operator on L2 defined by
ρX
L f := K(t, )f(t)dρ (t), f L2 .
K · X ∈ ρX
ZX
Let φ , j N bean orthonormal basis for L2 consisting of eigenfunctions of L with the correspond-
j ∈ ρX K
ing eigenvalues λ λ , j N. The assumption f ran(Ls ) implies that
j ≥ j+1 ∈ ρ ∈ K
∞
f = λsa φ
ρ j j j
j=1
X
for some h = ∞ a φ in L2 . In order to make use of the space constructed in the last section, our
j=1 j j ρX
last requirement is that K satisfies that span K(,x) :x X = C(X) and condition (2.8).
P { · ∈ }
Let cz,λ be a minimizer of (1.3) and let fz,λ be given by (1.6). For the minimization problem (1.3),
the hypothesis error and regularization error have the specific forms
(z,λ,g) := ( z(fz,λ)+λ fz,λ B) ( z(g)+λ g B),
P E k k − E k k
(λ,g) := (g) (f )+λ g ,
ρ B
D E −E k k
where g is a function in to be carefully chosen.
B
The use of the space enables us to discard the hypothesis error immediately.
B
Lemma 3.1 Under the above assumptions on K, there holds (fz,λ) (fρ) (z,λ,g) + (λ,g)
E −E ≤ S D
for all g .
∈B
Proof: By Theorem 2.4,
fz,λ = argmin z(f)+λ f B.
f∈BE k k
As a consequence, (z,λ,g) 0, which together with inequality (1.7) completes the proof. 2
P ≤
We next estimate the regularization error.
Lemma 3.2 If 0< s < 1 then
gin∈Bf D(λ,g) ≤ (khkL2ρX +khk2L2ρX)λ12+ss. (3.3)
If s 1 then f and
ρ
≥ ∈ B
(λ,f ) (λs−1 h )λ. (3.4)
D ρ ≤ 1 k kL2ρX
Proof: Firstly, we have for each ϕ L2 that L ϕ and by the Cauchy-Schwartz inequality that
∈ ρX K ∈ B
L ϕ = ϕ ϕ . (3.5)
K B L1 L2
k k k k ρX ≤ k k ρX
8
If s 1 then f = L ϕ where
ρ K
≥
∞
ϕ = λs−1a φ .
j j j
j=1
X
As λ is non-increasing,
j
∞ 1/2
ϕ λs−1 a 2 = λs−1 h .
k kL2ρX ≤ 1 | j| 1 k kL2ρX
(cid:18)j=1 (cid:19)
X
We then get by the above equation and (3.5) that
(λ,f ) =λ f λ ϕ λλs−1 h ,
D ρ k ρkB ≤ k kL2ρX ≤ 1 k kL2ρX
which is (3.4).
1
Suppose now that 0 < s < 1. If λ1 λ1+s then by (1.5),
≤
∞
D(λ,0) = E(0)−E(fρ) = kfρk2L2ρX = λ2jsa2j ≤ λ12+sskhk2L2ρX,
j=1
X
1
which implies (3.3). If λ1 > λ1+s then since λj decreases to zero as j tends to infinity, there exists
some N N such that λN+1 < λ1+1s λN. Put
∈ ≤
N
ϕ:= λs−1a φ .
j j j
j=1
X
It follows from (1.5) and (3.5) that
(λ,L ϕ) L ϕ f 2 +λ ϕ .
D K ≤ k K − ρkL2ρX k kL2ρX
We estimate that
N 1/2 N 1/2
λkϕkL2ρX = λ a2jλj2s−2 ≤ λλs1−+1s a2j ≤ λ12+sskhkL2ρX
(cid:18)j=1 (cid:19) (cid:18)j=1 (cid:19)
X X
and that
∞ ∞
kLKϕ−fρk2L2ρX = λ2jsa2j ≤ λ12+ss a2j ≤ λ12+sskhk2L2ρX.
j=N+1 j=N+1
X X
Combing the above two inequalities leads to (3.3). The proof is complete. 2
2s
We remark that the estimated regularization error in [20] was of the order O(λ2+s) for 0 < s 2.
≤
Turning to the sampling error, we follow the approach in [20] to decompose it into the sum
(z,λ,g) = (z,λ,g)+ (z,λ) where
1 2
S S S
1(z,λ,g) = ( z(g) z(fρ)) ( (g) (fρ)), 2(z,λ) = ( (fz,λ) (fρ)) ( z(fz,λ) z(fρ)).
S E −E − E −E S E −E − E −E
The first summand (z,λ,g) can be bounded by using the law of large numbers. By the same
1
S
arguments as those in [5, 20], we use the estimate in Lemma 3.2 to obtain an improved bound.
9
Lemma 3.3 Suppose that the output of sample data is bounded by a positive constant almost surely.
If 0 < s < 1 then for each ε > 0 there exists some g such that for all 0 < δ < 1, we have with
∈ B
confidence 1 δ that
− 2
2(s−1) 2s−1
λ 1+s λ 1+s 2
(z,λ,g) C + log
1 1
S ≤  m √m  δ
 
for some positive constant C . If s 1 then (z,λ,f ) = 0.
1 1 ρ
≥ S
For (z,λ), we cite the following result from [20].
2
S
Lemma 3.4 Suppose that (3.1) and (3.2) hold true. If λ 1 then we have with confidence 1 δ that
≤ − 2
S2(z,λ) ≤ 12(E(fz,λ)−E(fρ))+C2log 2δ +lλo2g(1+m)m−1+1η/α
for some positive constant C .
2
CombiningLemmas3.1,3.2,3.3,and3.4,wereachanewlearningrateestimateoftheℓ1-regularized
least square regression.
Theorem 3.5 Suppose that X satisfy (3.2), the output is bounded by a positive constant almost
surely, and f ran(Ls ) for some s > 0. Let K be a positive-definite reproducing kernel satisfying
ρ ∈ K
span K(,x) : x X = C(X), the condition (2.8) and the Lipschitz condition (3.1). Then there
{ · ∈ }
−1 1 1+s
exists some constant C > 0 such that with the choice λ = m 21+η/α1+2s, we have for all 0 < δ < 1
with confidence 1 δ that
−
− s 1 2+2m
(fz,λ) (fρ) Cm 1+2s1+η/α log , when 0 < s < 1 (3.6)
E −E ≤ δ
and
− 1 1+m
(fz,λ) (fρ) Cm 3(1+η/α) log , when s 1.
E −E ≤ δ ≥
Proof: We only discuss the case when 0 < s < 1 as the other situation is easier and can be shown in
a similar way. We choose λ = m−θ, θ > 0 and get by Lemmas 3.1, 3.2, 3.3, and 3.4 that there exists
some constant C >0 such that with confidence 1 δ
−
2+2m
(fz,λ) (fρ) Cm−γlog , (3.7)
E −E ≤ δ
where
1 2θ(1 s) 1 1 2s 2θs
γ = min 2θ,1 − , − θ, .
1+η/α − − 1+s 2 − 1+s 1+s
(cid:26) (cid:27)
The maximum of γ is achieved when
1 1 1+s
θ = .
21+η/α1+2s
Substituting the above choice into (3.7) yields (3.6). 2
Improvements of the learning rate can be achieved if higher regularity is imposed on the kernel K
[24] or better estimates of thesampling error are engaged [11]. Another remark is that the assumption
of positive-definiteness and symmetry on K might be abandoned by using the strategy in [20].
10