Control Capacity of Partially Observable Dynamic Systems in Continuous Time

0.6 MB·

by  Stas Tiomkin

1
Control Capacity of Partially Observable
Dynamic Systems in Continuous Time
Stas Tiomkin, Daniel Polani, Naftali Tishby
Abstract—Stochasticdynamiccontrolsystemsrelateinaprob- control and the reachable set of states in terms of information
abilistic fashion the space of control signals to the space of theory. For this, we consider stochastic dynamic systems as
corresponding future states. Consequently, stochastic dynamic
informationchannel,[19],betweenthesetwosets,ormorepre-
systems can be interpreted as an information channel between
cisely, between two random variables - the stochastic control,
the control space and the state space. In this work we study
7 this control-to-state informartion capacity of stochastic dynamic and the resulting state of dynamic system, [20]. Each of the
1
systems in continuous-time, when the states are observed only reachable set states is achievable by some control signal from
0
partially. The control-to-state capacity, known as empowerment, theadmissiblecontrolstate.Aselaboratedbelow,animportant
2
was shown in the past [1]–[6] to be useful in solving various
property of the correspondence between the reachable and
n Artificial Intelligence & Control benchmarks, and was used
the admissible sets is a number of different control signals
a to replace problem-specific utilities. The higher the value of
J empowerment is, the more optional future states an agent may which are distinguishable at a particular reachable set state.
8 reach by using its controls inside a given time horizon. A larger value would indicate a more influential control set.
1 The contribution of this work is that we derive an efficient Thisinfluenceiscastinthelanguageofinformationtheoryas
solution for computing the control-to-state information capacity
a question of the capacity between the control signal and the
foralinear,partially-observedGaussiandynamiccontrolsystem
] future state of the dynamic system.
T in continuous time, and discover new relationships between
I control-theoreticandinformation-theoreticpropertiesofdynamic
. systems.Particularly,usingthederivedmethod,wedemonstrate
s A. Empowerment
c thatthecapacitybetweenthecontrolsignalandthesystemoutput
[ doesnotgrowwithoutlimitswiththelengthofthecontrolsignal. This control capacity, also known as empowerment, [1]–
1 Tcohnitsrimbueatenssetfhfeacttiovnellyyttohethneeacro-nptarsotl-wtoin-sdtoawteocfapthaecitcyo,nwtrhoillesimgnoastl [6], has been shown to worked well as a universal heuris-
v oftheinformationbeyondthiswindowisirrelevantforthefuture tic in many contexts by generalizing ”controllability” in an
4 stateofthedynamicsystem.Weshowthatempowermentdepends information-theoreticsense.Itprovidesaplausibleframework
8 on a time constant of a dynamic system. for generating intrinsic (self-motivated) behaviour in agents,
9
eliminating the need for task-predefined, external rewards.
4 Index Terms—Information capacity, dynamic control systems,
0 empowerment, Gaussian process, Lyapunov equation, controlla- As an intuitive example, consider the underpowered pendu-
. bility gramian, stability, water-filling. lum, [5], driven by a low-energy stochastic control process.
1
It has a) stable and b) unstable equilibria positions, (the
0
7 I. INTRODUCTION bottom and the top position, respectively). In the bottom
1 REcentadvancesintheunderstandingofcomplexdynamic state, fewer states can be controllably reached than in the
: upright pendulum state. In the top state, while, on its own,
v systems reveal intimate connections between informa-
i tion theory and optimal control, [11]–[17]. To act optimally, unstable, with the help of a control signal, a richer set of
X
separate ”futures” (in a still precisely to define sense) can
engineering and biological systems must satisfy not only
r be controllably produced than in the bottom. The maximum
a the requirements of optimal control, (e.g., getting close to a
number of distinguishable controls at the system output is
target state, tracking smoothly a nominal trajectory etc.), but
givenbytheinformationcapacitybetweenthecontrolrandom
increasinglyotherimportantconstraints,suchasthelimitation
variable and the future state, viewed as a random variable
of bandwidth, the restriction of memory or limited delays.
depending on the (random) control signal. Here the dynamic
A stochastic dynamic control system is driven from state
system comprises the information channel.
to state by appropriate control signals from the admissible
set of control signals. All the system states achievable by Among other uses, it constitutes an example for agent con-
some admissible control signal define the reachable set of a trollersimplementingso-called”intrinsicmotivation”principle
dynamic system. In this work we take an alternative view of [7]–[10], a class of control algorithm which has recently
reachability, by studying the relation between the admissible receivedsignificantattention;controllersbasedontheseprinci-
ples substitute problem-specific utilities by generic measures
The Rachel and Selim Benin School of Computer Science and En- only depending on system dynamics to produce ”situation-
gineering,The Hebrew University, Givat Ram 91904, Jerusalem, Israel,
relevant” controls.
[email protected]
School of Computer Science, University of Hertfordshire, Hatfield AL10 The suitability of the control-state capacity (empowerment)
9AB,UnitedKingdom,[email protected] to implement an ”intrinsic motivation”-style controller was
The Edmond and Lilly Safra Center for Brain Sciences and The Rachel
explored in a series of theoretical and practical studies [1]–
andSelimBeninSchoolofComputerScienceandEngineering,TheHebrew
University,GivatRam91904,Jerusalem,Israel,[email protected] [5].
2
The empowerment landscape is computed for a dynamical oftheempowermentformalism.Onequestionwhichregularly
system and an agent is driven along this landscape as to arisesishowtochoosethishorizon.Ourstudy,amongstother,
maximize the local empowerment gradient. Typical effects shows that a characteristic time horizon may emerge through
include(butarenotlimitedto)theagentbeingledtounstable the dynamics only.
equilibriaandbeingstabilizedthere.Thecontrol-statecapacity Communication-limited control is another important con-
produces pseudo-utility landscapes from the dynamics of the cept in modern engineering where the control-state capac-
system and is a is a promising approach in designing a priori ity is useful is communication-limited control. The control-
(pre-task) controls for artificial agents. state capacity considers the channel between the controller
Until now, little work has been done in the domain of to the future state, while most of the work dealing with
continuous space and none in fully continuous (rather than communication-limited control, e.g. [11], [12], is focused on
discretized) time. Here, actions need to be constrained by the channel between the state sensor to the controller.
power, and, up to the present paper, no relation between For the first channel, the central question is: what is the
favoured time horizon and control signal power in the con- minimal information from the state to the controller that is
tinuoustimewasknown.Herewecontributeby1)calculating requiredinordertosatisfyacontrolobjective?.Forthesecond
empowermentforcontinuoustimeandspace(continuoustime channel, the central question is its dual: what is the maximal
wasnotconsideredinthepast),2)computingthefullsolution information that can be transferred from the controller to the
for the linear control case, 3) link time horizon characteristics state?”.
and power. These questions are complementary questions within the
To evaluate the empowerment of a partially observable dy- action-perception cycle paradigm, addressing the sensing and
namicsystemmeanstocomputethechannelcapacitybetween the actuation aspects of the cycle, respectively, [16]. Obvi-
thecontrolandthefinalstate,i.e.themaximumofthemutual ously, a comprehensive study of the control under communi-
information. We propose to compute the maximum of the cation constraints needs to consider also the dual, controller-
mutual information between the control process trajectory, to-state channel.
which we denote by u(0 → T) or briefly by u(·), and the These are only a selection of a multitude of directions
resulting system output, y(T), where the system dynamics where the control-state capacity has direct implications and
is perturbed by the process noise η, and the system output provides new understandings. In this paper we concentrate
is perturbed by the sensor noise ν. The relation between on studying the intrinsic properties of this channel, relating
theinformation-theoreticpropertieswithpropertiesofoptimal
control. In particular, we derive the relation between the
u(0→T) x˙(t)=f(x(t),u(t),η(t)) y(T)
energy, the time-horizon, and the capacity of the control-state
y(t)=g(x(t),u(t),ν(t))
of the dynamic system.
Thepaperisorganizedasfollowing.InSectionIIwedefine
Fig.1. Thedynamicsystem,f(·),isdrivenbytheuncontrollednoiseη(t), the problem of the information capacity between the control
and the stochastic control u(0 → T) to the future state x(T), which is signal and the future output. The solution is based on the bi-
partially observed by y(T) through the sensor g(·) within the observation
orthogonal representation of Gaussian process. We establish
noiseν(t).
the optimality conditions, KKT-conditions, for the optimal
variancesoftheGaussianprocessexpansion,andweshowthat
the control trajectory, u(0 → T) and the future observed
the optimal set of the variances can be found efficiently by an
state, y(T) is is thus stochastic and u(·) and y(T) are jointly
iterativewater-fillingalgorithm.Thesecondcomponentofthe
distributed according to Fig. 2. Empowerment of a stochastic
solution is formed by the expansionfunctions of the Gaussian
process expansion. We derive an optimality condition for the
u(0→T) y(T) expansion functions. This optimality condition is satisfied by
p(y(T)|u(0→T))
the spectral decomposition of the controllability Gramian of
thelinearcontrolsystem.Intherestofthepaperweelaborate
on the details of the solution.
Fig. 2. An information channel, given by the conditional probability
Particularly, In Section III we review briefly the bi-
distribution, p, which is induced by the dynamic system, f, and the sensor,
g,inFig.1. orthogonalexpansionofGaussianprocesses.InSectionIVwe
define the mutual information for continuous-time linear con-
trolsystems.InSectionVwedefinetheoptimizationproblem,
dynamical system is given by
which solves the problem defined in Section II. In Section VI
C∗ = maxI[u(·);y(T)] (1) we elaborate on the optimality conditions of the optimization
p(u(·)) problem, where we derive the KKT-optimality conditions and
Apart from extending the applicability of the formalism to set the conditions for the optimality of the Gaussian process
continuous time and arbitrary linear systems, the present expansionfunctions.InSectionVIIweprovidetheasymptotic
analysis contributes new insights to the multifaceted marriage analysis of the control-to-output capacity. In Section VIII we
between information theory and optimal control. provide computer simulations of the developed method and
Empowerment, in its original definition, requires the speci- explaintheirresults.And,finally,inSectionIXwesummarise
ficationofitstimehorizon.Thisistheessentialfreeparameter the work.
3
MAINRESULT matrix,Bn×p,andtheprocessnoisegainmatrixGn×n,scaling
the white Gaussian process noise, η(t), which has a given
We explore the dependency of the control capacity on
autocorrelation function:
the time horizon, T, and the power, P. We show how to
compute efficiently the information capacity, C(P,T), be- R (t ,t )=σ δ(t ,t ). (5)
η 1 2 η 1 2
tween the control signal and the future output in continuous-
time linear control systems. The solution is based on the bi- whereδ(t1,t2)isthedeltafunction,andση isthenoisepower.
orthogonal expansion of the Gaussian process. This enables The sensor noise, ν(t), is assumed to be the white Gaussian
us to decompose the solution procedure into two independent noise with the autocorrelation function:
aspects:the’iterativewater-filling’algorithm,andthespectral
R (t ,t )=σ δ(t ,t ). (6)
ν 1 2 ν 1 2
decomposition of the controllability Gramian.
In our example, we find particularly that the control input TheoptimizationtakesplaceoverthespaceoftheGaussian
signal contributes to the empowerment value only during a process probability distributions, P, or, equivalently, over the
temporally limited phase. Beyond this limited time window, space of the autocorrelation functions of the control process,
(cid:104) (cid:105)
the future output is not essentially influenced by the control R (t ,t ) = E u(t )u(cid:48)(t ) , because a Gaussian process is
u 1 2 1 2
signal.Thisfindingisconsistentwithintuition,asmemoryless defined uniquely by its autocorrelation function. We assume
linear control systems ’forget’ far past input events. theprocessnoise,η(t),tobeindependentofthecontrolsignal,
We establish a quantitative relation between empowerment u(t). We specifically do not restrict the control process to be
andtheintrinsicfeaturesofdynamicsystems,suchasthetime a stationary process.
constant,τ.Particularly,weshowthecontrolcapacityachieves The mutual information between continuous variables
afinitelimitforτ →0.Whilethisis,onfirstsight,weexplain would be unbounded unless their power is limited. To render
this effect mathematically, and provide a physical intuition. thequestionwell-defined,wethereforelookforthemaximum
We argue that various results of this work will permit of the mutual information, the capacity, under the constraint
the study of nonlinear control systems as well by locally of a given maximum total2 power of the control process:
approximating them by linear systems.
max I[u(·);y(T)] (7)
Ru(·,·)
NOTATION s.t. x˙(t)=Ax(t)+Bu(t)+Gη(t), (8)
The following notation is used in the paper. Uppercase and y(t)=Cx(t)+Fν(t), (9)
lowercaselettersdenoterealmatricesandvectors,respectively. (cid:40) (cid:41)
(cid:90) T
X(cid:48) denotes the (real-values) transpose of X. x[:i] is the i-th Tr Ru(t,t)dt ≤P. (10)
column of matrix X, Tr{·} denote the matrix trace and E[·] 0
denotes the expectation operator. The vertical bars,|·|, denote
The optimization over the space of the autocorrelation func-
the matrix determinant. The notation x , and x mean all the
(cid:98)i i tions corresponds to an optimization over the space of
components of x excluding the i-th component, and the i-th
symmetric-positive definite functions,
component of x, respectively.
We always assume the initial state of the dynamical system ∀t ,t ∈[0,T]:R (t ,t )=R (t ,t ), (11)
1 2 u 1 2 u 2 1
to be given and known, and thus, by abuse of notation, we (cid:90) (cid:90)T T
willneverwriteexplicitlytheconditioningwithrespecttothis ∀h∈L2([0,T]): Ru(t1,t2)h(t1)h(t2)dt1dt2 ≥0, (12)
initial state. 0 0
which introduce an uncountable set of constraints to the
problem in (7). We propose to perform the optimisation in
II. PROBLEMDEFINITION
(7) with regards to the components of the bi-orthonormal
Consider the following constrained optimization problem: expansionofGaussianprocessinsteadofR (·,·),asdescribed
u
in the next section.
max I[u(·);y(T)] (2)
p(u(·))∈P The capacity in (7), C, depends on the control power
s.t. x˙(t)=Ax(t)+Bu(t)+Gη(t), (3) constraint, P, the time horizon, T, and the dynamic system
matrices,A,B,G,andF.Thegoalofthisworkistocompute
y(t)=Cx(t)+Fν(t), (4)
this capacity efficiently, and to explore its properties with
where I[·;·] is the mutual information between the n- regard to the free parameters of the problem, and the ensuing
dimensional vector xn×1(T) at a given time T, and the p- properties of the dynamical control system.
dimensional Gaussian control process1, u (·), over times
p×1
t ∈ [0,T], distributed according to the probability density
III. CONTROLPROCESSREPRESENTATION
function, p(u(·)), which is restricted to the space of Gaussian
process distribution functions, P. The dependency of x(T) In this next section we represent the control process us-
ing the bi-orthogonal expansion of Gaussian processes, [21],
on u(·) is restricted by the dynamic affine-control system
whichhelpstorecasttheconstraintsin(11)and(12)inamore
with the dynamics matrix, A , the control process gain
n×n
convenient form for the optimization.
1inthispaperweconsiderw.l.og.zero-meanGaussianprocesses,because
themeandoesnotaffectthemutualinformation. 2apointwisemaximalpowermightalsobeconsidered.
4
It can be shown that any zero-mean Gaussian process u(t) and (9), given by:
may be represented by an appropriate choice of {gi(t)}∞i=1, (cid:90) T (cid:90) T
where {g (t)}∞ is a countable set of real orthonormal func- y(T)=C eA(T−t)Bu(t)dt+C eA(T−t)Gη(t)dt+Fν(T)dt
i i=1
tions and ui is a sequence of independent Gaussian random 0 0 (20)
variables.
Particularly, under the noise independence assumption:
Then, any zero-mean3 Gaussian process, u(t), may be rep-
resented by an appropriate choice of {gi(t)}∞i=1, and {ui}∞i=1 Σx(T)=Σu(T)+Ση(T), (21)
as following: (cid:48)
Σ (T)=CΣ (T)C +Σ (T), (22)
y x ν
∞
(cid:88)
u(t)= u g (t). (13) where
i i
i=1 T T
(cid:90) (cid:90)
Thisrepresentationisknownasthebi-orthogonalexpansion Σ (T)= eA(T−τ1)BR (τ ,τ )B(cid:48)eA(cid:48)(T−τ2)dτ dτ ,
u u 1 2 1 2
of Gaussian process. It is bi-orthogonal because the expan-
sion functions, {g (t)}∞ , are orthogonal, and the random 0 0
variables {ui}∞i=1 iare sit=at1istically independent. Using the bi- Σ (T)=(cid:90)T(cid:90)TeA(T−τ1)GR (τ ,τ )G(cid:48)eA(cid:48)(T−τ2)dτ dτ ,
orthogonalrepresentation,wecanrepresentthep-dimensional η η 1 2 1 2
control process as following. 0 0
Σ (T)=Tσ , (23)
(cid:80)  ν ν
u g (t)
i i1 i1
. where the last term is due to the sensor noise, which is not
(cid:126)u(t)= .  . (14)
 .  convolved with the system dynamics, but rather expands for
(cid:80)
u g (t)
i ip ip p×1 T time units as an unconstrained Wiener process with the
variance σ . We will denote by Σ the total covariance of
We assume that the components of the control process vector, ν n
(cid:126)u(t), areindependent.Consequently, theautocorrelation func- uncontrollednoise.Underthenoiseindependenceassumption,
tion of (cid:126)u(t) is diagonal, whose kk-th entry is specified by the the total uncontrolled covariance is the sum of the process
parameters σ of the control process as follows: noise and the sensor noise:
ik
(cid:88) Σ (T)=. CΣ (T)C(cid:48) +Σ (T), (24)
[R (t ,t )] = σ g (t )g (t ). (15) n η ν
u 1 2 kk ik ik 1 ik 2
i which is also the covariance of y(T), conditioned on the
Consequently, the power constraint in (8) appears as: control process, u(·):
Tr(cid:40)(cid:90) T R (t,t)dt(cid:41)= (cid:88)p (cid:88)σ ≤P, (16) Σy|u(·)(T)=Σn(T), (25)
u im
0 m=1i=1 which holds, because, knowing the control process, the only
and, the following orthogonality constraints on the function uncertainty in y(T) is due to the noise. Following the uncon-
set {g (t)} are: trolled noise covariance definition in (24), the future observ-
im im
able state covariance matrix in (22) is:
(cid:90) T
∀m,i,j : gim(t)gjm(t)dt=δij. (17) Σy(T)=CΣu(T)C(cid:48) +Σn(T). (26)
0
Using the definitions of the control and the noise autocorrela-
IV. MUTUALINFORMATION
tion functions given by (5) and (15), respectively, we have:
The mutual information between continuous variables is
denefitrnoepdy,oef.go.n[e19v]a,ribayblteh,ehd(·i)ffearnednctehebectownedeitniotnhael ddiiffffeerreennttiiaall Σu(T)= (cid:88)p (cid:88)σim(cid:90) T(cid:16)eA(T−t1)bmgim(t1)dt1(cid:17)
entropy of this variable with respect to the other, h(·|·). For m=1 i 0
Gaussian distributions, these are functions of the covariances (cid:16)(cid:90) T (cid:17)(cid:48)
· eA(T−t2)b g (t )dt , (27)
of the corresponding random variables. In our case, we are m im 2 2
0
icnotnetrreosltesdigsnpaelcoifivcearllaytiinmethepemrioudtuaTl iannfdormthaetiroensublteitnwgeseenntshoer Σ (T)=σ (cid:90) T eA(T−t)GG(cid:48)eA(cid:48)(T−t)dt(cid:31)0. (28)
η η
signal at the end of that period: 0
(cid:104) (cid:105) (cid:104) (cid:105) (cid:104) (cid:105) where bm = B[:,m]. The following notations will be used in
I u(0→T);y(T) =h y(T) −h y(T)|u(0→T)
the paper:
(cid:16)(cid:12) (cid:12)(cid:17) (cid:16)(cid:12) (cid:12)(cid:17)
=ln (cid:12)(cid:12)Σy(T)(cid:12)(cid:12) −ln (cid:12)(cid:12)Σy|u(·)(T)(cid:12)(cid:12) (18) w (t)=. eA(T−t)b , (29)
A,bm m
=ln(cid:16)(cid:12)(cid:12) Σy(T) (cid:12)(cid:12)(cid:17). (19) . (cid:90) T
(cid:12)Σy|u(·)(T)(cid:12) zim(T)= wA,bm(t)gim(t)dt, (30)
0
where Σy(T), and Σy|u(·)(T) are directly found, [22], from . (cid:90) T (cid:48)
the solution4 to the linear dynamic system equations in (8) W(A,bm,T)= wA,bm(t)wA,bm(t)dt(cid:31)0, (31)
0
3ameancanbeaddedifdesired having dimensions n × 1, n × 1, and n × n, respectively.
4weassumew.l.o.gzeroinitialcondition,whichcanbeabsorbedintothe Equation (31) is the controllability Gramian of the dynamic
processnoisecovariance.
5
system. We assume the system is controllable, which means reduces to the following5:
W(A,bm,T) is of full rank, [22]. (cid:12)(cid:12)Σ (T)+(cid:80)p (cid:80) σ Cz z(cid:48) C(cid:48)(cid:12)(cid:12)
Following the full representation of the control process max ln(cid:12) n m=1 i im im im (cid:12) (39)
in (13), we introduce a partial control signal representation (cid:126)σ,{gim(·)}im |Σn(T)|
without the im-th control component: p
(cid:88)(cid:88)
subject to σ =P, (40)
. (cid:88) im
u (·)= u (·), (32)
i(cid:99)m im m=1 i
j,k(cid:54)=i,m ∀i,m: σim ≥0, (41)
(cid:90) T
where, by abuse of notation, we write briefly,
∀i,m: g (t)g (t)dt=δ , (42)
im jm ij
. 0
uim(·)=uimgim(·), (33) .
where (cid:126)σ is a shortcut notation for (cid:126)σ = {σ } . The
im im
where uim is as in (14). To improve the readability we will corresponding unconstrained Lagrangian, L, is:
omit sometimes the time dependence, denoting z (T) by
im
zim.Usingtheabovenotations,thecontrolprocesscovariance L=L[(cid:126)σ,{gim(·)}im,λ,{νmij}mij,{γim}im], (43)
matrix in (27) appears as
where λ, and ν , γ are the Lagrange multipliers for
mij im
(cid:88)p (cid:88) (cid:48) the power, (16), the orthogonality constraints, (17), and the
Σ (T)= σ z (T)z (T). (34)
u im im im variance positivity constraints, respectively. The Lagrangian
m=1 i becomes:
Consequently, the final state covariance matrix appears as: (cid:12)(cid:12)Σ (T)+(cid:80)p (cid:80) σ Cz z(cid:48) C(cid:48)(cid:12)(cid:12)
(cid:12) n m=1 i im im im (cid:12)
Σx(T)= (cid:88)p (cid:88)σimzim(T)zi(cid:48)m(T)+Ση(T). (35) L=ln |Σn(T)| 
m=1 i (cid:32) p (cid:33) p
(cid:88)(cid:88) (cid:88)(cid:88)
The control process in (14) is represented by the collection −λ σ −P − γ σ
im im im
of the independent components, {uim}im. While a particular m=1 i m=1 i
tchoemcpoolnleecnttioinsdoefncootemdpuoinmen.tWweitwhoilultadthoeptpathrteicnuoltaarticoonmupi(cid:99)omnefonrt − (cid:88)p (cid:88)νm,i,j(cid:18)(cid:90) T gim(t)gjm(t)dt−δij(cid:19) (44)
uim. It will be seen useful to consider the partial control m=1 ij 0
process covariance matrix:
The corresponding KKT optimality conditions are:
(cid:48)
Σui(cid:100)m(T)=Σu(T)−σimzim(T)zim(T). (36) ∀m,i,t : δL[gim](t)=0, (45)
Consequently, the final state covariance, conditioned on the δgim
im-th control process component, is: ∂L
∀m,i : =0, (46)
∂σ
Σx|uim(T)=Σui(cid:100)m(T)+Ση(T), (37) (cid:90) Tim
which will be denoted in the following by Σ . The ∀m,i,j : g (t)g (t)dt=δ , (47)
x|uim im jm ij
equation (38) follows from the linearity and the independence 0
between different control process components. Intuitively, ∀m,i : γ σ =0, γ ≥0, σ ≥0, (48)
im im im im
knowing a particular control component u reduces the
im p
overall uncertainty in x(T) to the uncertainty due to u and (cid:88) (cid:88)
the noise. The final observable state covariance, condii(cid:99)mtioned σim =P. (49)
on the im-th control process component, is given by: m=1 i
where (48) is the complementarity KKT-condition. The opti-
(cid:48)
Σ (T)=CΣ (T)C +Σ (T), (38)
y|uim ui(cid:100)m n mizationproblemin(39-42)isaconvexoptimizationproblem
The mutual information given in (18) would be unbounded with regard to {σim}im for a given set of the expansion
withoutfurtherassumptionson{σim}im and{gim(t)}im.The functions, {gim(t)}im, which can be solved numerically by
assumption we use here is that the total sum of {σ } is the existing convex optimization solvers, [24]. For example,
im im
bounded by a positive constant P, and that the {g (t)} it can be found iteratively by the coordinate ascent algorithm,
im im
are continuous over the closed interval [0,T]. With these startingfromanarbitraryparametersset,{σim},andclimbing
constraints given, we proceed to formulate the optimization each time along different direction until convergence to the
problem in the next section. global maximum, which is guaranteed for any starting point,
because the objective function in (39) is concave. However,
the properties of the mutual information enable to derive a
formal solution for {σ } , as explained below.
im im
The objective function, the mutual information in (39), can
V. OPTIMIZATIONPROBLEM bedecomposedintoindividualcomponents,eachofwhichex-
presses the mutual information between a particular Gaussian
control process component and the future state. We prove this
Applying the particular expressions for the mutual infor-
in the next section. This separation provides an insight to the
mation in linear dynamic systems, (18), and the expressions
for energy and orthogonality constraints of the control pro-
cess constraints, (16) and (17), the optimization problem (7) 5weomittheconstantfactorsthatdoesn’taffectthemaximization.
6
nature of the optimal control signal, and will enables us to Consequently,
derive an implicit solution for {σim}im. (cid:32) (cid:33)
1 1
σ =max 0, − , (57)
Remark n Section.VI-C we show that, despite having intro- im λ z(cid:48) C(cid:48)Σ−1 ((cid:126)σ)Cz
ducedtheexpansionofgoveriasinfinite,onlyafinitenumber im y|uim im
of z-vectors (namely n) are in fact required to satisfy the whereΣy|uim((cid:126)σ)accordingto(38)is,(wehereexplicitlywrite
optimality condition of the Lagrangian (43) with regard to (cid:126)σ to make the dependence on the σ parameters explicit, and
drop writing the dependency on T)
{g (·)} . Therefore, in Eq. (34), we can limit ourselves to
im im
consider just the summation over i=1,..,n. (cid:88) (cid:48) (cid:48)
Σ ((cid:126)σ)= σ Cz z C +Σ . (58)
y|uim jk jk jk n
jk(cid:54)=im
A. Information Decomposition Altogether with the power constraint (49) we get:
Themutualinformationin(39)canbedecomposedtoasum (cid:32) (cid:33)
of mutual information terms between different control signal (cid:88)max 0, 1 − 1 =P, (59)
components: im λ zi(cid:48)mC(cid:48)Σ−y|1uim((cid:126)σ)Czim
1 (cid:88)p (cid:88)n (cid:110) where the global ’water-line’, λ, is adjusted by linear search.
I[u(·);y(T)]= I[u (·);y(T)]+ (50)
np im A particular control process variance, σim, given by (57),
m=1i=1 (cid:111) depends on all the other variances through Σ−1 ((cid:126)σ). A
+I[u (·);y(T)|u (·)] , (51) y|uim
i(cid:99)m im solution to the equations above forms a unique optimum
because of concavity. The formal solution given by (57) lends
whereI[u (·);y(T)],andI[u (·);y(T)|u (·)]isthemu-
tual informimation between the imi(cid:99)m-th control siigmnal component itself to implement an iterative water-filling scheme, [25],
and the future observable state, and the mutual information in the form of a two-phase fixed-point iteration. One phase
between the control signal without its im-th component and updates the σ (57), the other adapts the water-line (59).
im
the future observable state, conditioned on the im-th control
Fig.3showsthetypicalconvergenceofthisiteration,’Iterative
component, respectively. The decomposition follows directly
from the chain rule for the mutual information:
Water-FillingByFixed-PointConvergence
I[U(cid:48),U(cid:48)(cid:48);Y]=I[U(cid:48);Y]+I[U(cid:48)(cid:48);Y |U(cid:48)]. (52) 0.45
Consequently, the Lagrangian in (44) is equivalent to 0.4
0.35
L= n1p (cid:88)p (cid:88)n (cid:110)I[uim(·);y(T)]+I[ui(cid:99)m(·);y(T)|uim(·)](cid:111) 0.3 Pdi=im1σi(n)=1
m=1i=1 n)0.25
(cid:32) p (cid:33) p ~σ(
(cid:88)(cid:88) (cid:88)(cid:88) 0.2
−λ σ −P − γ σ
im im im
0.15
m=1 i m=1 i
(cid:88)p (cid:88) (cid:18)(cid:90) T (cid:19) 0.1
− ν g (t)g (t)dt−δ . (53)
mij im jm ij 0.05
m=1 ij 0
0 1 2 3 4 5
The advantage of this Lagrangian representation is due n,iterationnumber
to the fact that the second term of the objective function,
I[u (·);y(T)|u (·)]doesnotdependonσ bydefinition, Fig.3. Demonstrationoftheconvergenceof(cid:126)σntotheleadingcomponents,
i(cid:99)m im im where n is an iteration number. The resulting non-zero components of the
while, in the first term, for all i,m σ is separated from all
im controlprocessachievesthecapacityforthegivensetofz-vectors.Thetotal
other coefficients, σ , which suggests an efficient way powerissettoP =1.
j,k(cid:54)=i,m
to compute the optimal parameters set, (cid:126)σ.
Water-Filling’scheme,forarandomlychosendynamicsystem
of dimension n = 10, given by an arbitrary positive definite
VI. OPTIMALITYEQUATIONS
matrix Σ (cid:31) 0, and an arbitrary set of z-vectors. The y-
n
A. Expansion Variances - Generalized Water-filling. axis shows the components of (cid:126)σ(n), the x-axis the number
ComputingtheordinaryderivativeoftheLagrangianin(53) ofiterations,n.Thealgorithmwasrun100timesfordifferent
with regard to σ , and equating it to zero we get: initial conditions with the same random system matrix. The
im
dashed-lines show the numeric solution found by an explicit
∀i,m : ∂L(σim) = zi(cid:48)mC(cid:48)Σ−y|1uimCzim −λ−γ , convex optimization solver in comparison.
∂σ 1+σ z(cid:48) C(cid:48)Σ−1 Cz im
im im im y|uim im
(54) B. Complementary condition for non-zero capacity
Duetothecomplementaryslacknessconstraint(48),wehave: Even though σim is independent of zim, it turns out that
there is a complementary condition on σ and z which
z(cid:48) C(cid:48)Σ−1 Cz im im
σ >0⇒ im y|uim im −λ=0, (55) must be satisfied in order to get a non-zero capacity. This
im 1+σ z(cid:48) C(cid:48)Σ−1 Cz condition is stated and proved in the following lemma.
im im y|uim im Lemma 6.1: IfC hasfullrank,and∃(i,m)suchthatσ >
σ =0⇒z(cid:48) C(cid:48)Σ−1 Cz −λ−γ =0, (56) im
im im y|uim im im 0, but zim =0, then I[u(·);y(T)]≡0.
7
Proof: Assume ∃(i,m) such that σ >0 and z =0. {g (t)}, respectively. However, to compute the capacity we
im im im
Σ−1 (cid:31)0impliesthatz =0iffz(cid:48) C(cid:48)Σ−1 Cz = do not actually need to find the optimal expansion functions
y|uim(·) im im y|uim(·) im
0. In this case (55) implies that if z(cid:48) C(cid:48)Σ−1 Cz = 0, explicitly, but can express the capacity exclusively in terms of
then λ=0. Consequently, im y|uim(·) im the set {zim}.
The following lemma reveals the connection between the
∀i,m:z(cid:48) C(cid:48)Σ−1 Cz =0⇒∀i,m:z =0, (60)
im y|uim(·) im im z-vectors and the controllability Gramian in (31). This re-
which means sult is important, because it enables to compute the capac-
(cid:88) (cid:48) (cid:48) (cid:88) (cid:48) (cid:48) ity efficiently, and it strengthens the interplay between the
σ Cz z C + σ Cz z C =0.
im im im im im im information-theoretic properties of the dynamic system and
i,m:σim>0 i,m:σim=0 its optimal control properties.
(cid:124) (cid:123)(cid:122) (cid:125) (cid:124) (cid:123)(cid:122) (cid:125)
=0 due to (60) =0 due to σim=0 Lemma 6.2: Choosing the zjm to the eigenvectors of the
(61) controllabilityGramiansatisfiesequation(31).Particularly,set
(cid:112)
Altogether it follows that, if ∃(i,m) : σim > 0 and zim = 0, ∀i∈{1,..,n}:zim(T)= ωim(T)vim(T), (67)
then
∀i>n:z (T)=0, (68)
im
I[u(·);y(T)]=0. (62)
where ω (T) and v (T) are the corresponding eigenvalues
im im
(cid:4) and normalized eigenvectors of the controllability Gramian,
This lemma will be useful in the next section, where we W(A,b ,T) given by (31). (cid:3)
m
derivethesetof’z-vectors’,{z } ,whichsatisfiestheKKT
im im
optimality conditions in (45-49).
Proof of lemma 6.2: . Equation (66) is satisfied when
C. Expansion Functions - ’Gramian Decomposition’.
at least one of the following cases hold: a) σ = 0, b)
im
In this section we derive the optimality conditions for the z = 0, c) W˜ = W(A,b ,T)−(cid:80) z z(cid:48) = 0 , d)
expansion functions, {gim(t)}, by computing the functional Cim(cid:48)Σ−1Cz ∈ Null(W˜). mW.l.o.g. thejrejemxisjtms at lena×stnone
derivative, [23], of the Lagrangian (45). Particularly, with test y im
functions φ(t)im: σim >0,otherwisethecapacityiszero.AccordingtoLemma
6.1,andassumingC hasfullrank,thecorrespondingz (cid:54)=0,
∀i,m : (cid:90)0T δLδg[gimim](t)φim(t)dt=(cid:20)dL[gimd+(cid:15) (cid:15)φim](cid:21)(cid:12)(cid:12)(cid:12) . athned,cacsoens’edq)’uethnetlye,nCtir(cid:48)eΣn−yu1lCl zspimace(cid:54)=o0f tWo˜o,isbeacasoulsuetiΣonyit(cid:31)mo 0(6.6I)n.
(cid:15)=0
(63) Inthecase’c)’thesolutionisadecompositionoftheGramian
into the sum of 1-rank matrices z z(cid:48) :
As shown below, the optimality conditions for {gim(t)} en- im im
abletoderivethecorresponding’z-vectors’,{zim}im,defined W(A,b ,T)=(cid:88)z (T)z(cid:48) (T). (69)
in (30). The straightforward computation of the variation in m jm jm
(63) gives6: j
∀m,i,t : δL[gim](t)=σ Tr(cid:110)Σ−1Cz w(cid:48) (t)C(cid:48)(cid:111) The decomposition of a positive definite matrix to a sum of
δg im y im A,bm 1-rank matrices is not unique, and each of them solves (66).
im
+σ Tr(cid:110)Σ−1Cw (t)z(cid:48) C(cid:48)(cid:111) A special case is the eigenvalue decomposition:
im y A,bm im
(cid:48)
+2(cid:88)ν g (t)=0, (64) W(A,bm,T)=Vm(T)Λm(T)Vm(T) (70)
mij jm
n
j (cid:88) (cid:48)
= ω (T)v (T)v (T) (71)
im im im
which, due to the symmetry of the trace, Tr(A) = Tr(A(cid:48)),
i=1
and Σ−1 reduces to
y with ∀m : ω (T) > 0, and V(T)V(cid:48)(T) = I . All
im n×n
σimTr(cid:110)Σ−y1CzimwA(cid:48) ,bm(t)C(cid:48)(cid:111)=−(cid:88)νmijgjm(t). (65) ωim(T)>0isbecausethecontrollabilityGramianisassumed
j to be of full rank. Choosing the zjm as eigenvectors of
W(A,b ,T) completes the proof. (cid:4)
∀m,i and t∈[0,T]. As shown in Appendix A, the equation m
(65) is equivalent to the following equation
 
σimW(A,bm,T)−(cid:88)zjmzj(cid:48)mC(cid:48)Σ−y1Czim =0, (66) D. ’Water Filling with Gramian Decomposition’
j
Combining Lemma 6.2 with (57), the water-filling ex-
which will be useful in order to derive the optimal set of ’z-
pression for the expansion variances appears as, for i ∈
vectors’, {z }. It is worth to mention while the variation in
im {1,..,n},m∈{1,..,p}:
(63) is computed with respect to the expansion functions, and
(cid:32) (cid:33)
1 1
6theJacobi’sformula, ddtlog|Σy(t)|=Tr(cid:26)Σy(t)−1 ddtΣy(t)(cid:27),isused σim =max 0,λ − ωim(T)vi(cid:48)m(T)C(cid:48)Σ−y|1uim((cid:126)σ)Cvim(T) ,
(72)
inthederivation.
8
where the ’water-line’ is adjusted to satisfy the total power completely defines the capacity. The next section deals with
constraint, the unstable systems, where we show that the asymptotic
n p capacity is finite as well.
(cid:88) (cid:88)
σ =P. (73) 2) Unstable Systems: In this section we show that the
im
i=1m=1 asymptoticcapacityisfiniteaswell,ifalltheeigenvaluesofA
are positive. We need the following lemma in the asymptotic
The capacity will in general depend on the time horizon, T.
analysis of the capacity, C(∞), in systems where all the
In the following section, we show that the capacity, C(T), is
eigenvalues of A are positive.
limited by a finite value for T →∞:
Lemma 7.1: The following noise and control Gramians,
lim C∗(T)=C∞ <∞. (74)
T→∞ Σ˜ (A,G,T)=σ (cid:90) T e−AtGG(cid:48)e−A(cid:48)tdt, (82)
And, the capacity is linear in time for T →0: η η
0
(cid:90) T
lim C∗(T)≈cT, (75) W˜(A,b ,T)= e−Atb b(cid:48) e−A(cid:48)tdt, (83)
T→0 m m m
0
where c is a positive constant which is derived from the
define the same mutual information objective in (39), as the
properties of the dynamic system.
original Gramians, Σ (A,G,T) and W(A,b ,T) do.
n m
Proof:Fortheoptimalsetof{z }in(67)andunderthe
im
VII. ASYMPTOTICANALYSIS perfectsensorassumptionabove,theobjectivein(39)appears
as:
A.TIhnefintiottealtimunecohnotrriozollned noise covariance matrix, Σn(T), in ln(cid:32)(cid:12)(cid:12)(cid:12)(cid:12)In×n+Σ−η1(A,G,T)(cid:88)p W(A,bm,T)(cid:12)(cid:12)(cid:12)(cid:12)(cid:33), (84)
(cid:12) (cid:12)
(24) diverges for T → ∞, due to the unlimited growth of m=1
the sensor noise, Σ (T) (23), with time, T, which means where Σ (A,G,T)−1 and W(A,b ,T) can be represented
ν η m
that the channel capacity would go to zero. More refined by:
slotaotpemcaesnetsaroenptohsesibchlea,ninfeolncealpimaciittsyofnoerseTlf→toa∞peirnfetchteseonpseonr-, Ση(A,G,T)−1 =(cid:18)(cid:90) T eA(T−t)GG(cid:48)eA(cid:48)(T−t)dt(cid:19)−1 (85)
i.e. considering the special case y(t) = x(t). We found that 0
empowerment is finite both in stable systems, where all the =(cid:18)eA(cid:48)T(cid:19)−1(cid:18)(cid:90) T e−AtGG(cid:48)e−A(cid:48)tdt(cid:19)−1(cid:16)eAT(cid:17)−1
eigenvalues of A are negative, and in unstable systems, where 0
all the eigenvalues of A are positive. =(cid:18)eA(cid:48)T(cid:19)−1Σ˜ (A,G,T)−1(cid:16)eAT(cid:17)−1, (86)
η
1) Stable Systems: In this section we provide the asymp-
toticanalysisofthestablesystems.Whenthesystemisstable, and
tGhreanmtihane),paroncdetshsencooinsetrovlacroianntcroellambailtirtiyxGinram(2i8a)n, i(nth(e31n)oaisree W(A,bm,T)=(cid:16)eAT(cid:17)(cid:90) T e−Atbmb(cid:48)me−A(cid:48)tdt(cid:18)eA(cid:48)T(cid:19) (87)
0
finite for T →∞ and we can write: (cid:16) (cid:17) (cid:18) (cid:48) (cid:19)
Σ (A,G,∞)=σ (cid:90) ∞eA(T−t)GG(cid:48)eA(cid:48)(T−t)dt, (76) = eAT W˜(A,bm,T) eA T . (88)
η η
0 Consequently, the expression in (84) is equivalent to:
(cid:90) ∞
W(A,bm,∞)= 0 eA(T−t)bmb(cid:48)meA(cid:48)(T−t)dt, (77) ln(cid:32)(cid:12)(cid:12)(cid:12)(cid:12)In×n+Σ˜−η1(A,G,T)(cid:88)p W˜(A,bm,T)(cid:12)(cid:12)(cid:12)(cid:12)(cid:33), (89)
(cid:12) (cid:12)
and,theycanbefoundanalytically,(seefordetailse.g.,Theo- m=1
rem6.1in[22]),bysolvingthecorrespondingcontinuous-time (cid:4)
Lyapunov equations: When the system is unstable (i.e. all eigenvalues of A are
positive), then the controllability Gramians in (90) and (91)
(cid:48) (cid:48)
AΣ (A,G,∞)+Σ (A,G,∞)A =−GG , (78)
n n for T → ∞ can be found analytically by solving the corre-
(cid:48) (cid:48)
∀m:AW(A,b ,∞)+W(A,b ,∞)A =−b b , (79) sponding continuous time Lyapunov equations:
m m m m
(cid:88)n (cid:48) AΣ˜ (A,G,∞)+Σ˜ (A,G,∞)A(cid:48) =GG(cid:48), (90)
W(A,b ,∞)= ω (∞)v (∞)v (∞). (80) η η
m im im im
i=1 ∀m:AW˜(A,bm,∞)+W˜(A,bm,∞)A(cid:48) =bmb(cid:48)m, (91)
In this case, the water-filling solution is given by:
and,the’water-filling’solutionin(81)iscomputedwithW =
σ (∞)= W˜, and Σ =Σ˜ .
im n n
(cid:32) (cid:33)
1 1
=max 0, − ,
λ ω (∞)v(cid:48) (∞)Σ−1 (∞)v (∞) B. Infinitesimal time horizon
im im y|uim im
(81)
For infinitesimal times, empowerment becomes a linear
This provides the asymptotic capacity, C(∞), which is finite function of the horizon T, with a constant that depends on
for any finite power power constraint, P, where the noise various parameters of the system. In particular, for T → 0,
variance matrix, Σ (A,G,∞), and the controllability matrix empowerment vanishes again.
n
9
Particularly, the linearized noise covariance matrix in (24) For the systems shown in Fig.4, we compute the empow-
appears as: erment landscapes, C(P,T), for T∈[0,10] seconds, and P∈[0,10]
Σ (T (cid:28)1)=(cid:18)σ CeA0GG(cid:48)eA(cid:48)0C(cid:48) +σ I(cid:19)T Watts, which are shown at Fig.5. It is seen that empowerment
n η ν is a monotonic function of the power for a given time, and it
(cid:16) (cid:48) (cid:48) (cid:17) has a distinct maximum, (bright red region in the picture) as
= σ CGG C +σ I T, (92)
η ν functionofT.Thetimeofempowermentmaximumdecreases
which follows from the approximation of the integral in (28) with the decrease of the time constant, τ.
for T (cid:28) 1. Similarly, the control process covariance matrix The physical interpretation of this effect is that only a
(27) for T (cid:28)1 is: limited time window of the input control signal contributes
p n to the input-output mutual information. While, beyond this
Σu(T (cid:28)1)=(cid:88)(cid:88)σimeA0bmgim(0)eA(cid:48)0b(cid:48)mgim(0)T2 (93) time window there is no effective contribution to the capacity.
m=1i=1 Otherwise, the capacity would have grown without limit with
(cid:32) p n (cid:33)
= (cid:88)(cid:88)σ b g (0)b(cid:48) g (0) T2, (94) the length of the control signal. This time window depends
im m im m im
on the correlation decay time of linear control system. To
m=1i=1
which is quadratic in time. Consequently, the capacity for
T (cid:28)1 appears as
C(T (cid:28)1)=ln(cid:12)(cid:12)In×n+(Σn(T (cid:28)1))−1Σu(T (cid:28)1)(cid:12)(cid:12) (95)
=ln|I +M T|≈Tr(M)T, (96)
n×n n×n
where M=(cid:16)σηCGG(cid:48)C(cid:48)+σνI(cid:17)−1(cid:18) (cid:80)p (cid:80)n σimbmgim(0)b(cid:48)mgim(0)(cid:19)
m=1i=1
is a constant matrix.
VIII. SIMULATIONS
Using the developed method, we demonstrate by computer
simulations the relationship between empowerment and the
intrinsic properties of linear dynamic systems.
In the simulation we run the system for the fixed duration
of T seconds, however, we apply control only for the first t
seconds where t ∈ [0,T]sec. That way, one can identify at
Fig.5. TheempowermentlandscapesforthesystemsshownatFig.4.The
which time scales a control signal has effect on the observed blue and the red colors correspond to the low, (C →0 bits), and the high,
behaviour. (C∼3bits),valuesoftheempowerment,respectively.
The systems studied here were specified via their pole-zero
zoom in into the empowerment landscape we consider the
map and are shown in Fig.4. Each system, (marked by a
horizontal and the vertical slices of C(P,T) at P =5W,
distinct color), is given by three pairs of the conjugate poles,
and T =5sec, respectively. As seen in Fig.6, the graphs of
({p1,p†1},{p2,p†2},{p3,p†3}),andthezero,z,attheorigin.Thepoles C(P,T) become more similar to each other when the time
of each system are located at increasing distances from the
constant,τ,decreases.Toelucidatethiseffectweparametrized
imaginary axis, which is equivalent to decreasing the time
empowermentbythetimeconstant,τ,andsimulatedC (P,T)
constant, τ, of the system. The time constant is an intrinsic τ
for a range of the time constant.
propertyofadynamicsystem,whichreflectsthespeedandthe
amplitude of the input signal propagation through the system.
Increasing distances from the imaginary axis is equivalent to C(P,T0=5.0) vs. P C(P0=5.0,T) vs. T
increasing the time constant, τ, of the system. 2.5 2
s) s)
bit 2 bit1.5
Pole-ZeroMap StepResponse ( (
−1sec() 000...468 pp12 pp12 pp12 pp12pp12pp12 00..45 τττ ===310...036ssseeeccc CP,T()01.15 CP,T()0 1 ττ ==32..00sseecc
maginaryAxis−−000...0242 ppp3†3†2 ppp3†3†2 ppp3†3†2 ppp3†3†2ppp3†3†2ppp3†3†2 zzzzzz y(t) 000...0123 0.5 2 P4(wa6tt) 8 10 0.5 2 T4 (se6cττττ)====10008....3964sssseeeecccc10
I−0.6 p†1 p†1 p†1 p†1p†1p†1 −0.1 Fig. 6. The horizontal and the vertical slices of C(P,T) from Fig.5 at
−0.8 T =5secandP =5W,respectively.
−3 −2 −1 0 0 2 4 6 8 10
RealAxis(sec−1) t(sec)
As shown in Fig.7, C (P,T) converges to the finite value,
τ
Fig.4. Leftplot:Zero-polemapsofthecontinuous-timedynamicsystems Cτ(T)∼1.98bits for τ → 0. This behaviour, (the conver-
of order 6. Right plot: the step response of the systems in red, green, and gence to a finite capacity for decreasing τ), is qualitatively
blue with the corresponding time constants, τ = 3sec, τ = 1.3sec, and similar for different orders of dynamic systems and for differ-
τ =0.6sec,respectively. ent power, P. It is notable that the same value is achieved
10
when τ → 0, independently of the overall control time. to the imperfect sensor) futures can be selected by the control
Mathematically, the ratio of the noise Gramian to the control plant.
Gramian in (84) converges to a constant for τ →0:
The results of this work elucidate the effect of the finite
p p
lim Σ−1(A ,G,T)(cid:88)W(A ,b ,T)=σ ·(GG(cid:48))−1· (cid:88)b b(cid:48) .length of the input history, which holds the relevant informa-
η τ τ m η m m
τ→0 tion about the future. This is demonstrated by the constant
m=1 m=1
(97) capacity for growing T, as seen in Fig.6. We demonstrate this
using the example of stochastic dynamic systems, where we
The physical interpretation of this effect is as following.
computedexplicitlythecontrolcapacityanddemonstratedthe
The larger the distance of a pole from the imaginary axis
method by computer simulations.
is, the broader the frequency response, and simultaneously,
Theresultsofthisworkarederivedforlinearsystems.How-
the lower the gain at a corresponding frequency is. Moving
ever, it makes the approximative computation of this channel
the poles away from the imaginary axis is equivalent to
capacityaccessiblealsofornonlinearsystemsvialinearization.
increasingthesystemdamping(or,equivalently,decreasingthe
time constant, τ). There is a frequency window (as opposite Specifically,non-lineardynamicscanbeapproximatedlocally,
comprising a linear time-variant, (LTV), dynamics. However,
to the abovementioned time window) in which the input
in the case of LTV dynamics, one can not in general derive a
controlsignalthatcancontributetothevalueofempowerment,
closed-formanalyticexpression,asinthecaseofaLTIsystem.
when the damping of the system is increasing. Empowerment
achieves a finite value, as demonstrated in Fig.7, even if The established method for the computation of empow-
τ →0. erment in continuous-time linear control systems provides
important insights to the understanding of the information
2.5 Cτ(P=5.0,T) processing in linear control systems. Particularly, we show
that there is a limited time of control signal history which
contributes effectively to the input-output mutual information.
2
This time window defines an maximal length of the control
signal,wherethecontrolaffectthefutureoutputofthesystem.
(bits)1.5 The existence of a finite-sized effective time window for
Cτ
achieving the maximal capacity has a broader interpretation
1
T=0.10(sec) beyond the engineering interpretation for the systems: Opti-
T=0.60(sec)
T=1.10(sec) mized systems need, in given settings, only a limited history
0.5 T=1.60(sec)
TT==22..1600((sseecc)) inordertomakevaluablepredictions,requiredforthesurvival
T=3.10(sec)
of an organism or device and/or for the accumulation of a
101 100 10−1 10−2
τ(sec) value,[16]–[18].Inthisworkwedemonstratetheeffectofthe
length the past window on the example of stochastic dynamic
Fig. 7. Plot of Cτ(P,T). The x-axis is actually showing smaller τ as
progressingtotheright,(τ →0).The. dashed-linerepresentsCτ→0(P,T) system, and compute the capacity.
We show that increasing the damping (or, equivalently,
decreasing the time constant, τ,) of a continuous-time linear
control system does not result in zero empowerment. The
IX. SUMMARY
reason for a non-zero value of empowerment for τ →0 is the
In this work we explore the information channel between dependency of the mutual information on the Gramian ratio,
the control signal to the future outputs of a stochastic linear (84). The ratio achieves a finite value for τ →0. Technically,
control system. We derive an efficient method for computing we show that the value of empowerment in continuous-time
the maximal mutual information, the capacity, between the linear control systems can be computed without having to de-
controlsignalandthesystemfutureoutput.Thiscanbeviewed riveexplicitlythefunctionsetoftheGaussiancontrolprocess
assolvingthedualquestiontothepaperbyTatikonda&Mitter expansion, {g (t)} , ∀t ∈ [0,T]. Rather, it is sufficient to
i i
on control under communication constraints [11]. derive the ’z-vectors’, {z }, by the spectral decomposition
i
Thecapacityoftheinformationchannelbetweenthecontrol of the controllability Gramian. In particular, there may exist
and the future output, known as empowerment, was shown different sets of the expansion functions satisfying the set of
in earlier work to be useful in substituting problem-specific integral equations in (67) and (68). These degrees-of-freedom
utilities in established AI & Control benchmarks, [1]–[5]. may be exploited for designing dynamic control systems with
We develop an efficient method to estimate the value of different energetic and/or complexity constraints, on the ex-
empowerment in continuous-time linear stochastic control pansion function of the control signal. The optimal variances,
systems. We explore empowerment in partially observable {σ } ,ofthebi-orthonormalexpansionoftheGaussianprocess
i i
systems, where the distinguishable control (actuator) signals (13) can be found efficiently by the iterative water-filling
are evaluated from the imperfect sensor signal. Empowerment procedure. The following direction of this work is a design
can be - among other - interpreted as how well the control of algorithms based on the empowerment method for the
signalsaredistinguishableviatheimperfectsensorsatafuture control of continuous-time linear systems and its extension
time; or, equivalently, how many distinguishable (with respect to approximate empowerment in non-linear systems.