State Space representation of non-stationary Gaussian Processes

2.2 MB·

by  Alessio Benavoli

State Space representation of non-stationary Gaussian Processes
AlessioBenavoli MarcoZaffalon
[email protected] [email protected]
Dallemolleinstituteforartificialintelligence(IDSIA),
Manno,Switzerland
6
1
0 Abstract that the direct computation of the posterior of f is com-
2
putationally demanding. The computational cost is cubic,

n The state space (SS) representation of O(n3),inthenumberofobservations.ThismakesGPsun-
a
Gaussian processes (GP) has recently suitable for Big Data. Several general sparse approxima-
J
gainedalotofinterest.Themainreasonis tionschemeshavebeenproposedforthisproblem,(seefor
7
thatitallowstocomputeGPsbasedinfer- instance Quin˜onero-Candela and Rasmussen [2005] and
] ences in O(n), where n is the number of [RasmussenandWilliams,2006,Ch. 8]).
G
observations. Thisimplementationmakes Inthecasethefunctionf isdefinedonR,i.e.,x∈R,com-
L GPs suitable for Big Data. For this rea-
putational savings can be made by converting the GP into
s. son, it is important to provide a SS rep- State Space (SS) form and make inference using Kalman
c resentation of the most important kernels filtering. Notethatthecasex∈Risparticularlyimportant
[
usedinmachinelearning. Theaimofthis because it includes time-series analysis (x is time). The

1 paper is to show how to exploit the tran-
connectionbetweenGPsandSSmodelsiswellknownfor
v sientbehaviourofSSmodelstomapnon-
somebasickernelsandrecentlyithasgainedalotofinter-
4
stationarykernelstoSSmodels.
4 est. CertainclassesofstationaryCFscanbedirectlycon-
5 verted into state space models by representing their spec-
1 traldensitiesasrationalfunctions[Sa¨rkka¨andHartikainen,
0 1. Introduction
2012,Sarkkaetal.,2013,SolinandSarkka,2014]. More-
.
1 over,anexplicitlinkbetweenperiodic(non-stationary)CFs
0 In machine learning, Gaussian Processes (GP) are com- andSSmodelshasalsobeenderivedin[SolinandSa¨rkka¨,
6
monly used modelling tools for Bayesian non-parametric 2014].
1
inference [O’Hagan and Kingman, 1978, Neal, 1998,
: TheconnectionbetweentheSSrepresentationofGPsand
v MacKay, 1998, Rasmussen and Williams, 2006, Ras-
i GPsmodelsusedinmachinelearningisimportantformany
X mussen, 2011, Gelman et al., 2013]. For instance in GP
reasons. First,inmachinelearning,ithasbeenshownthat
regression,y =f(x),theaimistoestimatef from(noisy)
r GPs, with a suitable choice of the kernel, are universal
a observation y. A natural Bayesian way to approach this
function approximators [Rasmussen and Williams, 2006,
problem is to place a prior on f and use the observa-
Williams,1997]. Moreover,GPscanalsobeusedforclas-
tions to compute the posterior of f. Since f is a func-
sification, with only a slight modification. Second, infer-
tion, theGPisanaturalpriordistributionforf [MacKay,
ences in SS models can be computed efficiently by pro-
1998, Rasmussen and Williams, 2006]. A GP, denoted as
cessingtheobservationssequentially. Thismeansthatthe
GP(0,k (x,x(cid:48))),iscompletelydefinedbyitsmeanfunc-
θ computationalcostofinferenceintheSSrepresentationof
tion (usually assumed to be zero) and covariance function
GPsisO(n). Third,SSmodelsrepresentGPsthroughSto-
(CF) (also called kernel) k (x,x(cid:48)), which depends on a
θ chasticDifferentialEquations(SDE).Theyreturnamodel
vectorofhyperparametersθ. Bysuitablychoosingtheker-
that directly explains the time-series and not only fits or
nel function, we can make GPs very flexible and conve-
predictsit. ItiswellknownthatSDEsarebasicmodelling
nient modelling tools. However, a drawback with GPs is
toolsineconometrics,physicsetc..Hence,ifweareableto
map the most important kernels used in machine learning
to the SS representation we can “kill three birds with one
stone”,i.e.,wecanhaveanexplanatorymodelwithauni-
versal function approximation property at a cost of O(n).
Thisistheaimofthispaper. Inparticular,thegoalistoex- with ψ(t ,t ) = exp((cid:82)tkF(t)dt) is the state transition
k 0 t0
tendthework[Sa¨rkka¨andHartikainen,2012,Sarkkaetal., matrix,whichisobtainedasamatrixexponential.1
2013]byprovidingaSSrepresentationofthemostimpor-
Proposition 1. Assume that E[f(t )] = 0, then the vec-
tantkernelsusedinmachinelearning. 0
tor of observations [y(t ),y(t ),...,y(t )]T is Gaussian
1 2 n
Inparticular,wewillshowthatnon-stationarykernelscan distributed with zero mean and covariance matrix whose
bemappedintoSSmodelsbyconsideringthetransientbe- elementsaregivenby:
haviour. It is well known that the time response of lin-
E[y(t )y(t )]=
ear SS models is always the superposition of an initial- i j
conditionpartandadrivenpart.Theresponseduetoinitial- =C(ti)ψ(ti,t0)E[f(t0)fT(t0)](C(tj)ψ(tj,t0))T+
conditionsisoftenignored, becauseitvanishesinthesta- min(cid:82)(ti,tj)
tionary case (it is transient). However, for non-stationary h(ti,u)h(tj,u)q(u)du
systems, the transient never vanishes and, thus, it deter- t0
(3)
mines the behaviour of the system. Even with a zero ini-
where we have exploited the fact that E[dw(u)dw(v)] =
tialcondition,wecanhaveatransientbehaviourduetothe
q(u)δ(u − v)dudv and defined h(t ,t ) =
1 2
driven part. We will show that by taking into account the
C(t )ψ(t ,t )L(t )(h(...)iscalledimpulseresponse).
1 1 2 2
transient, we can map the linear regression, periodic and
spline kernel to SS models. Moreover, we will also study
The proof of this proposition is well known (see for in-
thetransientbehaviourforstationarysystemstoshowthat
stanceJazwinski[2007]),butwehavereportedthederiva-
in this case it vanishes. To reconcile these two cases, we
tions of this proposition (and next propositions/theorems)
will make use of the Laplace transform that is able to ac-
in appendix for the convenience of the reader. From
countforthetransientbehaviour. Thisisadifferencew.r.t.
(3), it is evident that, given the time-varying matrices
the work by Sa¨rkka¨ and Hartikainen [2012], Sarkka et al.
A(t),L(t),C(t), the CF of a LTV system is completely
[2013] where they employed the Fourier transform. Then
defined by: (i) the covariance of the initial condition
we will show how to map the neural networks kernels to
E[f(t )fT(t )];(ii)theCFofthenoiseE[dw(u)dw(u)].
SSmodels.Forthispurposewewilluselineartime-variant 0 0
SS,thatareintrinsicallynon-stationary. Finally,bymeans In case the SS model is Linear Time-Invariant (LTI), i.e.,
of simulations we will show the effectiveness of the pro- F(t) = F,L(t) = L,C(t) = C,q(t) = q,wecanusethe
posed approach and the computational advantages by ap- Laplacetransformtoderive(2)and(3)usingonlyalgebraic
plyingittolongtime-series.Inthiswork,forlackofspace, computations. The Laplace transform of a function x(t),
wewillassumethatthereaderisfamiliarwiththemachine definedfort≥0,is:
learningrepresentationofGPsandwewillonlydiscussthe ∞
(cid:90)
SSrepresentation. x(s)= e−stx(t)dt,
0
2. StateSpacemodel where the parameter s is the complex number s = σ +
ιω,withσ,ω ∈ Randιdenotingtheimaginaryunit. The
Laplace transform exists provided that the above integral
Letusconsiderthefollowingstochasticlineartime-variant
is finite. The values of s for which the Laplace transform
(LTV)statespacemodel[Jazwinski,2007]
existsarecalledtheRegionOfConvergence(ROC)ofthe
Laplacetransform.ByusingtheLaplacetransform,wecan
(cid:26)
df(t) = F(t)f(t)dt+L(t)dw(t),
rewritethedifferentialequationin(1)inanalgebraicform:
(1)
y(t ) = C(t )f(t ),
k k k
sf(s)−f(t )=Ff(s)+Lw(s), (4)
0
where f(t) = [f (t),...,f (t)]T is the (stochastic) state wheref(s),w(s)aretheLaplacetransformsoff(t),w(t).2
1 m
vector, y(t ) is the observation at time t , w(t) is a Sincey(s)=Cf(s),wehave
k k
one-dimensional Wiener process with intensity q(t) and y(s)=C(sI−F)−1f(t )+C(sI−F)−1Lw(s)
0
F(t),L(t),C(t)areknowntime-variantmatricesofappro-
=C(sI−F)−1f(t )+H(s)w(s), (5)
priatedimensions. Wefurtherassumethattheinitialstate 0
f(t0)andw(t)areindependentforeacht ≥ t0. Itiswell whereH(s) = C(sI−F)−1Liscalledthetransferfunc-
know that the solution of the stochastic differential equa- tion of the linear time-invariant (LTI) SS model and I is
tionin(1)is(seeforinstanceJazwinski[2007]):
1ThematrixexponentialiseA =I+A+A2/2!+A3/3!+....
(cid:90)tk side2rBinygdwefi(tn)inagstahedeLtearpmlaicneisttricanisnfpourtm. W(4e),uwseetahrieswnorotantgiolynoconnly-
f(t )=ψ(t ,t )f(t )+ ψ(t ,τ)L(τ)dw(τ), (2)
k k 0 0 k for convenience, but then we define the correct inverse Laplace
t0 transformin(6).
theidentitymatrix. SinceaproductintheLaplacedomain foreacht ,t ≥0,whichcorrespondstothelinearregres-
i j
correspondstoaconvolutionintime,itfollowsthat sionCF[RasmussenandWilliams,2006,Sec. 4.2.2].
y(t )=L−1(cid:0)C(sI−F)−1(cid:1)f(t )+(cid:90) tkh(t −u)dw(u),
k 0 k ThisconnectionbetweenSSandlinearregressioniswell-
t0 known,herewehaveshownhowtoderivetheCF.Higher
(6)
where L−1(·) denotes the inverse Laplace transform. By orderlinearregressionCFscanbeobtainedbyconsidering
computingE[y(ti)y(tj)],weobtainagain(3). Theoutput E[f(t0)fT(t0)]=diag([σ12,...,σm2 ])and
ofbothLTVandLTIsystemsisclearlycompletelydefined (cid:20) 0 I (cid:21) (cid:20) 0T (cid:21)
F= m−1 m−1 , L= m−1
by the SS matrices, the initial condition and the stochas- 0 0T 1
m−1
tic forcing term dw(t). The aim of the next sections is to (cid:2) (cid:3)
showthatbysuitablychoosingthesethreecomponentswe C= 0m−1 1 , (9)
can obtain SS models whose CF coincides with the main where0 isthe(m−1)zerovector.
m−1
kernelsusedinGPs.
3.2. PeriodicKernel
3. Non-stationaryCFsdefinedbyLTISS
withoutforcingterm ConsiderthefollowingLTISS:
(cid:20) (cid:21)
F= 0 ωk ,C=(cid:2) 1 0 (cid:3), (10)
Inthissection, wewillshowthattwoimportantCFsused −ω 0
k
inGPscanbeobtainedbytwoLTISSmodelswithoutsto-
chastic forcing term. Their output is therefore completely with q(t) = 0 for all t ≥ 0. By computing ψ(t,t0) =
determinedbytheinitialconditionsand,thus,theCFthey exp((t−t0)F),wehavethat
defineisnon-stationary(itdependsonti,tj ≥t0). (cid:20) cos(ω (t−t )) sin(ω (t−t )) (cid:21)
ψ(t,t )= k 0 k 0 ,
0 −sin(ω (t−t )) cos(ω (t−t ))
k 0 k 0
3.1. Linearregressionkernel
and,so
Assumewithoutlossofgeneralitythatt =0,
0 y(t )=Cψ(t ,t )f(t )
i i 0 0
(cid:20) (cid:21) (cid:20) (cid:21)
0 1 0
F= , L= , =cos(ωk(t−t0))f1(t0)+sin(ωk(t−t0))f2(t0)
0 0 1
=a cos(ω (t−t ))+b sin(ω (t−t )), (11)
(cid:2) (cid:3) k k 0 k k 0
C= 1 0 , (7)
wherewehavewrittenf (t )=a andf (t )=b .
q(t)=0∀t≥03andsothereisnotforcingterm(dw(t)= 1 0 k 2 0 k
0). ThiscorrespondstothefollowingLTISSmodel: Proposition3. ConsidertheSSmodelin(10)andassume
 df1(t) = f (t), that[ak,bk]areGaussiandistributedwithzeromean,vari-
 dt 2 ances E[a2], E[b2] and uncorrelated E[a b ] = 0. From
df2(t) = 0, (8) k k k k
 ydt(tk) = f1(tk). (w2i)t,hwzeerhoamveeathnaatn[yd(Ct1F),...,y(tn)]isGaussiandistributed
FromtheequationsofthisSSmodel,sincey(t)=f1(t)we E[y(t )y(t )]=E[a2]cos(ω (t −t ))cos(ω (t −t ))
1 2 k k 1 0 k 2 0
derivethatf (t) = f(t)(itisthefunctionofinterest)and
f (t) = df(1t) is its derivative. By computing ψ(t,t ) = +E[b2k]sin(ωk(t1−t0))sin(ωk(t2−t0)).
2 dt 0
exp((cid:82)tt0Fdt)=exp((t−t0)F),wehavethat =E[a2k]cos(ωk(t2−t1))
(cid:20) 1 t−t (cid:21) for each t ,t ≥ t , where last equality holds if E[a2] =
ψ(t,t )= 0 , i j 0 k
0 0 1 E[b2].
k
and, therefore, y(t ) = Cψ(t ,t )f(t ) = f (t )+(t−
i i 0 0 1 0
t )f (t ). ThisSSmodeldefinesaperiodicCF.However,itisevident
0 2 0
(seeinparticular(11))thatitcanonlyrepresentsinusoidal
Proposition 2. Consider the SS model in (8) and assume
type periodic functions. However, we know that any pe-
that f(t0) is Gaussian distributed with zero mean and co- riodicfunctionf(t)in[−pe,pe], thatisintegrable, canbe
variance E[f(t )fT(t )] = diag([σ2,σ2]) and t = 0.
0 0 1 2 0 approximatedbyFourierseries:
From(2), wehavethat[y(t ),...,y(t )]isGaussiandis-
1 k
J (cid:20) (cid:18) (cid:19) (cid:18) (cid:19)(cid:21)
tributedwithzeromeanandCF (cid:88) 2πkt 2πkt
f(t)≈ a cos +b sin , (12)
E[y(ti)y(tj)]=σ12+σ22titj, k=1 k pe k pe
3Sinceq(t) = 0,thematrixLiscompletelysuperfluous. We where ak,bk ∈ R are the coefficients of the Fourier se-
haveintroduceditonlybecausewewillusethismodellater. ries, whichdependonf(t), andJ ∈ Nistheorderofthe
approximation.4 Therefore, we can approximate any peri- 4. Non-stationaryCFsdefinedbyLTISS
odic function by a sum of J SS models of type (10) with withzeroinitialconditions
ω = 2πk. Forinstance,forJ =2,wecanconsidertheSS
k pe
model
In this section, we will show that an important CF used
  in GPs can obtained by a LTI SS model with stochastic
0 ω 0 0
1
forcingtermandzeroinitialconditions.
F= −ω1 0 0 0 ,C=(cid:2) 1 0 1 0 (cid:3).
 0 0 0 ω2 
0 0 −ω 0 4.1. Splinekernel
2
(13)
WederivetheCFforthecubicsmoothingsplines.Consider
Its transfer function φ(t ,t ) is a diagonal block matrix theLTISSmodelin(7), butthistimeassumethatq(t) =
i 0
withblocks 1∀t≥t0,E[f(t0)fT(r0)]=0andt0 =0,then
(cid:20) cos(ωk(t−t0)) sin(ωk(t−t0)) (cid:21), (cid:90)ti (cid:90)ti
−sin(ωk(t−t0)) cos(ωk(t−t0)) y(ti)= Cψ(ti,τ)L(τ)dw(τ)= (ti−τ)dw(τ)
0 0
fork =1,2. Hence,from(3),wehavethat
(16)
E[y(ti)y(tj)]= (cid:80)2 E[a2k]cos(ωk(ti−t0))cos(ωk(tj −t0))andso,from(2),E[y(ti)y(tj)] = (cid:82)0min(ti,tj)(ti−τ)(tj −
k=1 τ)dτ.
+E[b2]sin(ω (t −t ))sin(ω (t −t )),
k k i 0 k j 0
(14) Proposition4. ConsidertheLTISSmodelin(7), butthis
wherewehaveassumedthatE[f(t )fT(t )]isablockdi- timeassumethatq(t)=1∀t≥0andE[f(0)fT(0)]=0.
0 0
agonalmatrixwithblocks Then,from(2),wederivethat[y(t ),...,y(t )]isGauss-
1 n
iandistributedwithzeromeanandCF
(cid:20) E[a2] 0 (cid:21)
0k E[b2k] , (15) E[y(ti)y(tj)]=|ti−tj|min(t2i,tj)2 + min(t3i,tj)3,
(17)
fork = 1,2. InFourierseries,thefunctionf(t)isknown
and so the coefficients ak,bk can be computed based on foreachti,tj ≥0,whichisthekernelforthecubicsmooth-
f(t). InGPregression,wedonotknowf(t)andsowedo ingsplines[RasmussenandWilliams,2006,Sec. 6.3.1].
notknowa ,b . Wemustestimatethesecoefficientsfrom
k k
data.5 Thisisthereasonwehaveassumedapriordistribu-
HigherordersplinescanbeobtainedconsideringSSmod-
tiononthesecoefficients. Thisprioriscompletelydefined
els as in (9). The connection between SS and smoothing
by the variances E[a2],E[b2] for k = 1,2,...,J. If we
k k splines has been first derived in Kohn and Ansley [1987].
furtherassumethatE[a2]=E[b2]=q2thenwehaveonly
k k k Here, we have highlighted the connection with GPs and
J parameters to specify, the q . We can further assume
k computedtheCF(17).
that all the parameters are functions of a single parameter
(cid:96)≥0;andpenalizehighorderfrequencies. Forinstance,if
wechooseq2 =2exp(−(cid:96)−2)(cid:80)(cid:98)J−2k(cid:99) (2(cid:96)2)−k−i,itcanbe 5. StationaryCFsdefinedbyLTISSmodels
k i=0 (k+i)!i!
shownthat
A stationary CF is a CF that only depends on t −t . In
j i
(cid:88)J (cid:18)2πk (cid:19) this section we will present SS models whose CF corre-
E[y(t )y(t )]= q2cos (t −t )
i j k p j i spondstostationarykernelsusedinGPs. Sincestationary
e
k=0 CFs satisfy k(τ) = k(−τ) for τ = t −t , i.e., they are
j i
(cid:32) (cid:18)π(t −t )(cid:19)2(cid:33) even functions defined on R, it is convenient to introduce
−J−→−−∞→exp −2 sin j i ,
(cid:96)2 p thebilateralLaplacetransform:
e
∞
(cid:90)
which is the periodic CF used in GPs [Rasmussen and B(f(τ))= e−sτf(τ)dτ,
Williams, 2006, Sec. 4.2.3]. This result has been derived
−∞
bySolinandSa¨rkka¨[2014],herewehavehighlightedmore
extensively the transient analysis and the connection with thatisdefinedforfunctionsthattakevaluesinR.Whenthe
theFourierseries. ROCincludestheimaginaryaxis,thenfors = ιω,thebi-
lateralLaplacetransformreducestotheFouriertransform.
4Wecanalsoincludetheconstantterm(k=0)intheseries. Assume that the CF is stationary. The Wiener-Khintchine
5Whentheperiodpeisunknown,wecanestimateitfromdata. theorem[Chatfield,2013]statesthattheCFiscompletely
definedbyitsFouriertransform(itsbilateralLaplacetrans- 5.1. Mate´rnkernelforν =3/2
formcomputedfors=ιω)(whenitexists)andviceversa:
(cid:90) (cid:90) LetusconsideragainthestationaryMate´rnkernelforν =
k(τ)= Sy(ιω)e2πιωτdω, Sy(ιω)= k(τ)e−2πιωτdτ, 3/2,i.e.,E[y(ti)y(tj)]=k(τ)= 4λ13e−λ|τ|(1+λ|τ|)for
R R τ = t −t [RasmussenandWilliams,2006,Sec. 4.2.1].
2 1
where Sy(ιω) is the Fourier transform of k(τ) also called ThebilateralLaplacetransformofk(τ)is
thespectraldensity.
1
B(R(τ))= ,
Theorem 1 (Representation theorem). Define Sy(ιω) = (λ−s)2(λ+s)2
S (s) and assume that S (s) that is a (proper) rational
y y which is a rational function. We can then apply Theorem
functionofs. ThenthereexistsastableLTIsystemwiththe
1 and, in this case, H(s) = 1/(λ+s)2 and S (s) = 1.
impulseresponsehsuchthat w
Using(20)wecanderivetheSSmodel:
(cid:90) t
(cid:20) (cid:21) (cid:20) (cid:21)
y(t)= h(t−u)dw(u) (18) 0 1 0
F= , L=
−∞ −λ2 −2λ 1
where w is a stationary process with uncorrelated incre- (cid:2) (cid:3)
C= 1 0 , (21)
mentsandspectraldensityS (s)=q >0. IntheLaplace
w
domainthiscanbewrittenas q(t) = 1 ∀ t ≥ t . Its impulse response is h(t) =
0
texp(−λt)fort≥t .
S (s)=H(s)H(−s)q, (19) 0
y
whereH(s) = b(s)/a(s)isarationalfunctionwhosede- Proposition 5. Consider (21) and assume that
nominatora(s)hasallrootswithnegativerealparts,b(s) E[f(t0)fT(t0)] = 0. Then, from (2), we obtain that
hasnorootswithpositiverealpartandqisapositivereal [y(t1),...,y(tn)] is Gaussian distributed with zero mean
constant.6 andCF
E[y(t )y(t )]= e−λ|tj−ti|(1+λ|tj−ti|)
This is standard result for LTI systems. By interpreting i j 4λ3
q as the Bilateral Laplace transform of the noise dw(t) − e−λ(−2t0+ti+tj)(1+λ(−2t0+ti+tj)+λ2(t0−ti)(t0−tj))
4λ3
(S (s) = q),thenEquation(19)relatesthespectralfunc-
w
foreacht ,t ≥t .
tions of the output of a SS model described by the trans- i j 0
fer function H(s) with that of the input (the noise w(t))
through the transfer function of the SS model H(s). We Itcanbeobservedthatfort0 →−∞,thesecondtermgoes
canuse(19)toderivetheSSmodelthatisrelatedtoS (s). to zero and and we obtain the Mate´rn CFs for ν = 3/2
y (cid:112)
Thesearethesteps: (i)findthezeroesandpolesofS (s); andλ= (3)/(cid:96). OtherMate´rnCFsforν =3/2,5/2,...
y
(ii)takeallthepolesp withrealnegativepartandzeroes canbeobtainedviaLTISSmodelsinasimilarway. This
i
withnon-positiverealpart;(iii)decomposeH(s)as connectionbetweenLTISSmodelsandMate´rnkernelhas
(cid:81) been firstly discussed in Sarkka et al. [2013]. Here, we
H(s)= (cid:81)ii((ss−−pzii)) = bs0ns+n+a1bs1nsn−−1+1+·····+·+abnn−−11ss++abnn, hshaovwenretphoarttethdethCeFCbFecfoomrte0sfisntaittieo(nnaorny-fsotarttio0n→ary−c∞ase.)and
andderivethe(observablecanonical)LTISSmodel:
 0 1 0 ··· 0    5.2. SquareExponentialkernel
r
1
F= 00... 00... ·01·· ·.·.....· 001 ,L=rnr...−21 [LnReelatsEums[uyns(ostwei)nyc(oatnnjsd)i]dWe=riltlkhiae(mτs)tsa,=ti2o0ne0ax6rpy,(√−Ssqe2ucτ(cid:96)a.22r)e4fe.o2xr.p1oτ]n.e=nIttsitajlb−ikleatrti--
−an −an−1 −an−2 ··· −a1 rn eralLaplacetransformisSy(s)= 2π(cid:96)exp((cid:96)22s2),whose
C=(cid:2)1 0 0 ··· 0(cid:3), (20) ROC is Re(s) = 0 and, hence, s = iω. Sy(s) is not a
rational function, so we cannot directly apply Theorem 1.
where However,itisareal-valuedpositivefunction(itispositive
r   1 −1b  and non-zero) and it is analytic, so we can approximate
0 0
r1 a1 1  b1 1/Sy(s)withitsTaylorexpansioninzeroandobtain:
 ... =··· ...   ...  S (s)≈=√2π(cid:96)d!2d 1 .
y d!2d+d!2d−1(cid:96)2s2+···+(cid:96)2ds2d
r a ··· a 1 b
n n 1 n (22)
TheaboveLTIsystemhasCFk(τ).
WecanfindH(s)bydeterminingtherootsofthepolyno-
6SinceSy(s)isarealevenfunctions,thezeroesandpolesare mial at the denominator that have negative real part. This
symmetricw.r.t.therealaxisandmirroredintheimaginaryaxis. canbedonenumerically,butthenextresultisuseful.
Theorem 2. The roots of the denominator in (22) can be 6.1. Non-stationarySEkernel
obtainedbycomputingtherootsfor(cid:96) = 1andthendivid-
√
ingthemfor(cid:96). ThisgivesH(s),whileS (s)= 2π(cid:96)d!2d. LetusconsiderthisCF:
w (cid:96)2d
E[y(t1)y(t2)]=e−2σ1m2 t2ie−2σ1s2(ti−tj)2e−2σ1m2 t2j (24)
This result is useful because it allows us to compute off-
line the solutions of (22) even when the hyperparameter (cid:96) where σm2 ,σs2 are hyper-parameters. This CF is clearly
isunknown. Letusconsiderforinstancethecased=2 non-stationary. However,itscentraltermisthesquareex-
√ 1 ponential CF and, therefore, it can be approximated by a
Sy(s)≈8 2π(cid:96)ω4(cid:96)4+4ω2(cid:96)2+8. LTVSSthatisasimplevariantoftheLTISSthatdefines
√ thesquareexponentialCF.Forinstance,forσs = σm = 1
Therootsofthedenominatorfor(cid:96)=1ares=± 2±i2.7 thisLTVSSmodelis:
Hence, the roots corresponding to the stable part are (cid:20) (cid:21) (cid:20) (cid:21)
√ 0 1 0
− 2±i2≈−1.5538±i0.6436andso F= −(a2+b2) −2a , L= 1
1 (cid:104) (cid:105)
H(s)= , C= e−t2/2 0 , (25)
(s+a)2+(b)2
witha=1.5538andb=0.6436,whichcanberepresented The impulse response is h(t,u) =
bythefollowingLTISSmodel: e−t2/2exp(−a(t−u))sin(b(t−u)) foru<tandtheCF
b
(cid:20) (cid:21) (cid:20) (cid:21) √
F= 0 1 , L= 0 E[y(ti)y(tj)]=8 2πe−t21/2e−t22/2
−(a2+b2) −2a 1
C=(cid:2) 1 0 (cid:3), (23) min(cid:90)(ti,tj)exp(−a(ti−u))sin(b(ti−u))
√ b
with q(t) = 8 2π/(cid:96)3. The inverse Laplace transform of t0
H(s)ish(t)= exp(−at)sin(bt). exp(−a(t −u))sin(b(t −u))
b · j j du
b
Proposition 6. Consider (23) and assume that
E[f(t0)fT(t0)] = 0. Then, from (2), we obtain that This is the d = 2 approximation. For t0 → −∞, the in-
[y(t1),...,y(tn)] is Gaussian distributed with zero mean tegral term depends only on |t2 −t1|. This CF is used in
andCF neuralnetworksresearchandtheconnectionwithGPshas
√ beendiscussedbyWilliams[1997].
E[y(t )y(t )]=8 2π
i j
min(cid:90)(ti,tj)exp(−a(t −u))sin(b(t −u)) 6.2. LTVsystemdefiningtheneuralnetworkkernel
i i
b LetusconsiderthefollowingLTVSSmodel
t0
exp(−a(t −u))sin(b(t −u)) (cid:20) 0 e−tTΣ1/2r (cid:21) (cid:2) (cid:3)
· j j du F= ,C= 1 0 , (26)
b 0 0
for each ti,tj ≥ t0. For t0 → −∞, it only depends on with q(t) = 0∀t > 0, t = [1,t]T, r = [r0,r1]T and
|t2−t1|. Σ = diag(σ2,σ2)isadefinitepositivedefinitematrix. Its
0 1
transferfunctionis
ThisconnectionbetweenLTISSmodelsandsquaredexpo- (cid:34) √ (cid:35)
nentialkernelshasbeenderivedin[Sarkkaetal.,2013]and φ(ti,t0)= 1 π(erf(tiTΣ1/22rr1)σ−1erf(tT0Σ1/2r)) .
in[SarkkaandPiche´,2014]theyhavestudiedtheconver- 0 1
gencepropertyoftheTaylorseries. Here,wehavederived
Hence,wehavethat
thenewresultinTheorem2, computedtheCFforafinite
y(t )=Cψ(t ,t )f(t )
ts0tat(inoonna-rsytaftoiornt0ar→y c−as∞e)(asnedeasphpowennditxhafotrththeeCpFroboef)c.omes =fi1(t0)+ √πi(er0f(tiTΣ01/22rr1)σ−1erf(tT0Σ1/2r))f2(t0) (27)
and
6. LTVsystems E[y(t )y(t )]=
i j
=C(t )ψ(t ,t )E[f(t )fT(t )](C(t )ψ(t ,t ))T
i i 0 0 0 j j 0
Up to now, we have only worked with LTI SS models.
=erf(tTΣ1/2r)erf(tTΣ1/2r),
Hereafter, we will show that moving from LTI to LTV i j
(28)
allows us to map two fundamental non-stationary kernels
where we have assumed that f(t ) is Gaussian distributed
[Williams,1997]toSSmodels. 0
with zero mean and covariance matrix E[f(t )fT(t )] =
0 0
7Bydividingthemfor(cid:96)weobtaintherootsfor(cid:96)(cid:54)=1. [erf(tT0Σ1/2r),2√r12σπ1]T[erf(tT0Σ1/2r),2√r12σπ1]. Therefore,
from(27),itisevidentthat(26)modelserflikefunctions. 4 4
Wecanaddexpressivitytothemodelasfollows 2 2
 0 e−tTΣ1/2r1 0 0  0 0
 0 0 0 0  (cid:45)2 (cid:45)2
F= ,
 0 0 0 e−tTΣ1/2r2  (cid:45)4 (cid:45)4
(cid:45)4 (cid:45)2 0 2 4 (cid:45)4 (cid:45)2 0 2 4
0 0 0 0 4 4
√
(cid:2) (cid:3)
C= 1 0 1 0 / 2. 2 2
Its transfer function φ(t ,t ) is a diagonal block matrix 0 0
i 0
withelements (cid:45)2 (cid:45)2
(cid:34) √ (cid:35)
1 π(erf(tiTΣ1/2rk)−erf(tT0Σ1/2rk)) (cid:45)4(cid:45)4 (cid:45)2 0 2 4 (cid:45)4(cid:45)4 (cid:45)2 0 2 4
2r1kσ1 ,
0 1
FIGURE 1. Contour plot of the NN CF
fork =1,2. Hence,wehavethat for J = 15,30,50,∞ (from left to
right).
y(t )=
i
√12k(cid:80)=21f1k(t0)+ √π(erf(tiTΣ1/22rr1kk)σ−1erf(tT0Σ1/2rk))f2k(t0), 24 24
(29)
wheref(t )=[f (t ),f (t ),f (t ),f (t )]T andso 0 0
0 11 0 21 0 12 0 22 0
(cid:45)2 (cid:45)2
E[y(t )y(t )]=
i j
=C(t )ψ(t ,t )E[f(t )fT(t )](C(t )ψ(t ,t ))T (cid:45)4(cid:45)4 (cid:45)2 0 2 4 (cid:45)4(cid:45)4 (cid:45)2 0 2 4
i i 0 0 0 j j 0
2
= 1 (cid:80) erf(tTΣ1/2r )erf(tTΣ1/2r ) FIGURE 2. Contour plot of the NN CF
2 i k j k
k=1 using deterministic sampling for J =
(30) 10,20(fromlefttoright).
where we have assumed that E[f(t )fT(t )]
0 0
is a block diagonal matrix with elements
7. Scalingandmeasurementnoise
[erf(tT0Σ1/2rk),2r√12kπσ1]T[erf(tT0Σ1/2rk),2r√12kπσ1]. If
we increase the number of elements k and keep this
In the previous derivations, we have assumed that there is
block-diagonalstructure,wehavethat
notmeasurementnoise. Wecanaddameasurementnoise
E[y(t )y(t )]= 1 (cid:80)J erf(tTΣ1/2r )erf(tTΣ1/2r ). tothemeasurementequationasfollows
i j J i k j k
k=1 y(ti)=C(ti)x(ti)+v(ti), (33)
(31)
where v(t ) = N(0,σ2) is assumed to be independent to
i v
Theorem3. Assumethatthevectorsrk aresampledfrom the process noise and initial condition. Under these as-
astandardGaussiandistribution,then sumptions, for each CF = (...) computed in the above
J sections,wehavenowthat
E[y(t )y(t )]= 1 (cid:80) erf(tTΣ1/2r )erf(tTΣ1/2r )
i j J i k j k E[y(t )y(t )]=(...)+σ2.
k=1 i j v
−J−→−−∞→(cid:82) erf(tTΣ1/2r)erf(tTΣ1/2r)N(r;0,I)dr Moreover,sometimesitisdesirabletohaveapositivescal-
i j
(cid:18) (cid:19) ingparameterthatmultipliestheCF.Thiscanbeincluded
∝ 2 sin−1 √ 2tTiΣtj byrescalingeitherthestochasticforcingtermortheinitial
π (1+2tTiΣti)(1+2tTjΣtj) condition(forSSwithoutforcingterm). Finally, notethat
(32)
different kernels can be combined in an additive way, by
whichistheneuralnetworkCF[Williams,1997].
simply stacking SS models in block-diagonal matrices as
shownfortheperiodicorNNSSmodel.
Figure 1 shows the contourplot of the CF for different J.
It is evident that at the increase of J converges to the NN
CF.Toincreasetheconvergencespeed,wecanchoosethe 8. Estimates
vectorsr inadeterministicway(smartsampling).Wefol-
k
lowtheapproachproposedbyHuberandHanebeck[2008]. OncewehavedefinedtheSSmodelforagivenCF,weaim
Figure2showsthecontourplotoftheCFforthedetermin- toestimatethestateofthemodelgiventhemeasurements.
isticsampling. itisevidentthatforJ =20theapproxima- Given the hyperparameters of the CF, this can be carried
tion is already very good (compare it with the last plot in outinthreesteps:
Figure1relativetoJ =∞). (a) Discretize the continuous-time SDE to obtain a
cedi(pnbofi(fisg)xvcca((rir2Cteeik)tanoe.)ntm-|lcytyipe(mbtum1eyte)a,uSt.srDt.iihnxE.eg,.oyKfTp(tahtrhkolimsi)bs)aas,GbtnteihalpfiiautltysbtesiasirsaidiGnncegaanP.luslDisytsFyicaocnfna.unsnTiscbhtteseiocomnonemaa(pnPpupDatlneFydd-) (cid:45)(cid:45)1001....0550....................0...2.................0....4.................0...6..................0...8..................1..0 (cid:45).2............................(cid:45).1...................(cid:45).00000......11234.........................1........................2
msT(pcm(h)axeot(roitlxtkahs)Ciot|nyofgs(mtttteh1phpi)ues,tre.ePes.Dtt.ui,mFrynat(chtttaeenhnse)o)b.beestGatciiamnoTuemahsdtsepeibusayntmeotdefhaevtnhKeperoaayslnsmttedeafratfiienoccrofiigevlintavetrelriny.PanDbacFlyel (cid:45)(cid:45)1001....0550....................0...2.................0....4.................0...6..................0...8..................1..0 (cid:45).2............................(cid:45).1...................(cid:45).000000.......112345.........................1........................2
observations. The last two steps have both complexity
O(n). To estimate the hyperparameters of the CF, we FIGURE 3. Sawtooth inference with SS
adoptaBayesianapproach. Weplaceaprioronthehyper- periodicmodeloforderJ =2,6.Gauss-
parametersandthenweestimatethemusingaMonteCarlo ianinferencewithSSNNmodelforJ =
approach. In practice, we employ a Rao-Blackwellised 10,20.
particle filtering [Smith et al., 2013, Ch. 24]. We have
ra9Cep.oppnoeEsrnitxddedeiaxrm.tthhpeelfeosltsleoapwswintoogfotwtthhoefwuwnahcvotieloenasnindferGenacuesssicahneme in -1123......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................0...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................2..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................0............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................4................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................0............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................6......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................0...............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................8.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................1.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................0
ya(t)=S(10π(t+0.1))+va(t), yb(t)=N(4t,0,1)+vb(t) 4
3
with S(t) = t − (cid:98)t(cid:99), v (t) ∼ N(0,0.1) and v (t) ∼
a b 2
N(0,0.05). The first is a sawtooth wave with period 0.2.
1
ThesecondisaGaussiandensity. Wehavegenerated100
0.2 0.4 0.6 0.8 1.0
observation from the above regression model considering
-1
t ∈ [0 : 0.01 : 1]inthefirstcaseandrandomlygenerated
-2
t∼4Unif[0,1]−2inthesecondcase. Ourgoalistoem-
ploytheperiodicSSmodeland,respectively,NNSSmodel
FIGURE4. Sunspotsnumber
to estimate the two functions. The result is shown in Fig-
ure3.Inparticular,thetwopairedplotsreporttheposterior
themissingdata,thenumberofobservationsis58(cid:48)306.For
mean and relative credible region for two different orders
inferenceswehaveappliedtheperiodicSSmodelswiththe
of the periodic and NN SS model. Consider the sawtooth
goalofestimatingtheperiodofsunspotactivitycycle,that
case. The first plot shows the estimate based on the peri-
is about eleven years. Figure 4 reports the observations
odicSSmodeloforderone(onlyonecosinecomponent).
(the time has been normalized in [0,1] and the number of
This SS model can estimate exactly only periodic cosine-
sunspots have been standardized) and the posterior mean
typefunction,buttheunknownfunctionisaperiodicsaw-
computedusingtheperiodicSSmodelwithJ =8compo-
tooth wave. To improve the estimate, we can increase the
nents.Theestimatedperiodforthesunspotcyclewas10.36
number of cosine terms. For J = 6, it is evident that we
years. ThetimetoruntheSSmodelonthistimeserieswas
already obtain an accurate estimate of the sawtooth wave.
1.5hoursonstandardlaptopwithamodelimplementedin
AsimilarcommentholdsalsofortheNNSSmodel. Atthe
Matlab.
increasingoftheapproximationorder,theSSposteriores-
timategetsclosertothetruefunction(itbettermodelsthe
tailsoftheGaussianPDFthathasgeneratedthedata). 10. Conclusions
9.1. BigData: sunspotsnumber In this paper we have shown that, by exploiting the tran-
sient behaviour of SS models, it is possible to map non-
Finally, to show the computational efficiently of the pro- stationary Gaussian Processes (GP) kernels to state space
posed approach, we have applied the SS model for infer- models (SS). In particular, we have shown how to map
encesinalongtime-series. Inparticular, wehaveconsid- to SS models the neural network Kernels. This is impor-
eredthesunspottimeseries–adailytimeseriesthatreports tantbecauseSSmodelsallowstoreducethecomputational
the number of sunspots from 1820 to 2015. After treated complexityofinferenceswithGPsfromcubictolinearin
the number of observations. As future work, we plan to Simo Sarkka, Arno Solin, and Jouni Hartikainen. Spa-
continue to study the relationship between the SS model tiotemporal learning via infinite-dimensional bayesian
representation and GPs as well as we plan to extend this filtering and smoothing: A look at gaussian process re-
worktomultivariateregressionproblems. gression through kalman filtering. Signal Processing
Magazine,IEEE,30(4):51–61,2013.
References AdrianSmith,ArnaudDoucet,NandodeFreitas,andNeil
Gordon. Sequential Monte Carlo methods in practice.
Chris Chatfield. The analysis of time series: an introduc-
SpringerScience&BusinessMedia,2013.
tion. CRCpress,2013.
Arno Solin and Simo Sa¨rkka¨. Explicit link between pe-
AndrewGelman,JohnBCarlin,HalSStern,DavidBDun-
riodic covariance functions and state space models. In
son, Aki Vehtari, and Donald B Rubin. Bayesian data
Proceedings of the Seventeenth International Confer-
analysis. CRCpress,2013.
enceonArtificialIntelligenceandStatistics,volume33,
Marco F Huber and Uwe D Hanebeck. Gaussian filter
pages904–912,2014.
basedondeterministicsamplingforhighqualitynonlin-
Arno Solin and Simo Sarkka. Gaussian quadratures for
ear estimation. In Proceedings of the 17th IFAC World
state space approximation of scale mixtures of squared
Congress(IFAC2008),volume17,2008.
exponentialcovariancefunctions. InMachineLearning
Andrew H Jazwinski. Stochastic processes and filtering
forSignalProcessing(MLSP),2014IEEEInternational
theory. CourierCorporation,2007.
Workshopon,pages1–6.IEEE,2014.
Robert Kohn and Craig F Ansley. A new algorithm for
Christopher KI Williams. Computing with infinite net-
spline smoothing based on smoothing a stochastic pro-
works. Advancesin neuralinformation processingsys-
cess. SIAM Journal on Scientific and Statistical Com-
tems,pages295–301,1997.
puting,8(1):33–48,1987.
David JC MacKay. Introduction to Gaussian processes.
InBishop,C.M.,editor,NeuralNetworksandMachine
Learning,pages133–166,1998.
R. M. Neal. Regression and classification using gaussian
processpriors. IninBernardo, JMandBerger, JOand
Dawid,APandSmith,AFMeditors,BayesianStatistics
6: ProceedingsofthesixthValenciainternationalmeet-
ing,volume6,page475,1998.
A.O’HaganandJ.F.C.Kingman. Curvefittingandopti-
maldesignforprediction. JournaloftheRoyalStatisti-
cal Society. Series B (Methodological), 40(1):pp. 1–42,
1978.
JoaquinQuin˜onero-CandelaandCarlEdwardRasmussen.
Aunifyingviewofsparseapproximategaussianprocess
regression. TheJournalofMachineLearningResearch,
6:1939–1959,2005.
Carl Edward Rasmussen. The gaussian processes
web site, February 2011. URL http://www.
gaussianprocess.org/.
CarlEdwardRasmussenandCKIWilliams. Gaussianpro-
cessesformachinelearning.2006.TheMITPress,Cam-
bridge,MA,USA,38:715–719,2006.
Simo Sa¨rkka¨ and Jouni Hartikainen. Infinite-dimensional
kalman filtering approach to spatio-temporal gaussian
processregression. InInternationalConferenceonArti-
ficialIntelligenceandStatistics,pages993–1001,2012.
SimoSarkkaandRobertPiche´. Onconvergenceandaccu-
racyofstate-spaceapproximationsofsquaredexponen-
tialcovariancefunctions. InMachineLearningforSig-
nalProcessing(MLSP),2014IEEEInternationalWork-
shopon,pages1–6.IEEE,2014.
AppendixA. Proofs A.4. Proposition4
Inthiscase,wehavethath(t,u)=h(t−u)=(t−u)and
A.1. Proposition1 sofrom(2),
min(cid:90)(ti,tj)
Bydefinitionofcovariance,wehavethat
E[y(t )y(t )]= h(t ,u)h(t ,u)q(u)du
i j i j
t0
min(cid:90)(ti,tj)
E[y(ti)y(tj)]= (ti−u)(tj −u)du,
C(t )ψ(t ,t )E[f(t )fT(t )](C(t )ψ(t ,t ))T+
i i 0 0 0 j j 0 0
(cid:82)ti(cid:82)tjC(t )ψ(t ,u)L(u)E[dw(u)dw(v)] =|t −t |min(ti,tj)2 + min(ti,tj)3,
i i i j 2 3
t0t0
·LT(v)ψT(t ,v)CT(t )
j j wherewehaveexploitedthefactthatq(u)=1.
=C(t )ψ(t ,t )E[f(t )fT(t )](C(t )ψ(t ,t ))T+
i i 0 0 0 j j 0
min(cid:82)(ti,tj)C(t )ψ(t ,u)L(u)LT(u)ψT(t ,u)CT(t )q(u)du A.5. Proposition5
i i j j
t0
=C(t )ψ(t ,t )E[f(t )fT(t )](C(t )ψ(t ,t ))T+
i i 0 0 0 j j 0
E[y(t )y(t )]=
i j
min(cid:82)(ti,tj)
h(ti,u)h(tj,u)q(u)du min(cid:90)(ti,tj)
t0 (t −u)e−λ(ti−u)(t −u)e−λ(tj−u)du
i j
(34)
where we have exploited that E[dw(u)dw(v)] = t0
q(u)δ(u−v)dudvanddefinedh(t,u)=C(t)ψ(t,u)L(u). e−λ|tj−ti|(1+λ|tj−ti|)
4λ3
− e−λ(−2t0+ti+tj)(1+λ(−2t0+ti+tj)+λ2(t0−ti)(t0−tj))
4λ3
A.2. Proposition2 foreacht ,t ≥ t . Itcanbeobservedthatfort → −∞,
i j 0 0
thesecondtermgoestozero.
Sincey(t ) = Cψ(t ,t )f(t ) = f (t )+(t−t )f (t ).
i i 0 0 1 0 0 2 0
and f(t0) is Gaussian distributed with zero mean and co- A.6. Proposition6
variance E[f(t )fT(t )] = diag([σ2,σ2]), from (2), we
0 0 1 2
havethatfort =0: Theresultoftheintegralis
0
(cid:16) (cid:16)
e−a(−2min(ti,tj)+ti+tj) a(bsin(b(−2min(t ,t )+t +t ))
i j i j
E[y(t )y(t )]=E[(f (0)+t f (0))(f (0)+t f (0))] −acos(b(−2min(t ,t )+t +t )))+(cid:0)a2+b2(cid:1)cos(b(t −t ))(cid:17)
i j 1 i 2 1 j 2 i j i j i j
=σ02+titjσ12 (35) −e−a(−2t0+ti+tj)(cid:16)(cid:0)a2+b2(cid:1)cos(b(ti−tj))
foreacht ,t ≥0. (cid:17)(cid:17)
i j
+a(bsin(b(−2t +t +t ))−acos(b(−2t +t +t )))
0 i j 0 i j
(cid:0)4ab2(cid:0)a2+b2(cid:1)(cid:1)−1
A.3. Proposition3
The second term that multiplies e−a(−2t0+ti+tj) vanishes
fort →−∞andsowehave
0
TheproofissimilartothatofProposition2butconsidering
that e−a|tj−ti|(cid:16)a(bsin(b|t −t |)
j i
−acos(b|t −t |)+(cid:0)a2+b2(cid:1)cos(b|t −t |)(cid:17)
j i j i
y(t )=Cψ(t ,t )f(t ) (cid:0)4ab2(cid:0)a2+b2(cid:1)(cid:1)−1
i i 0 0
=cos(ω (t−t ))f (t )+sin(ω (t−t ))f (t )
k 0 1 0 k 0 2 0
wherewehaveexploitedthefactthat−2min(t ,t )+t +
i j i
(36)
t =|t −t |andcosineisanevenfunction.
j j i