Source Coding with in-Block Memory and Causally Controllable Side Information

0.31 MB·English

by  Osvaldo Simeone

Source Coding with in-Block Memory and Causally
Controllable Side Information
Osvaldo Simeone
New Jersey Institute of Technology
Email: [email protected]
Abstract—Therecentlyproposedset-upofsourcecodingwitha Xn W Xˆ =u (W,Yi)
sideinformation “vendingmachine”allows thedecoder toselect Enc Dec i i
actionsinordertocontrolthequalityofthesideinformation.The
actionscandependonthemessagereceivedfromtheencoderand A =v (W,Yi−1)
Y
onthepreviouslymeasuredsamplesofthesideinformation,and i i
i
arecostconstrained.Moreover,thefinalestimateofthesourceby
the decoder is a function of the encoder’s message and depends
P(yL ||aL |xL)
causally on the side information sequence. Previous work by
3
Permuter and Weissman has characterized the rate-distortion-
1
cost function in the special case in which the source and the
0
“vending machine” are memoryless. In this work, motivated by
2 Fig.1. Sourcecodingwithin-blockmemory(iBM)andcausallycontrollable
therelatedchannelcodingmodelintroducedbyKramer,therate- sideinformation.
n distortion-cost function characterization is extended to a model
a within-blockmemory.Variousspecialcasesarestudiedincluding
J block-feedforward and side information repeat request models.
n, as detailed below. The model under study is motivated by
5 Index Terms: Source coding, block memory, side information
channelcodingscenarioputforthin[2]andcanbeconsidered
“vending machine”, feedforward, directed mutual information.
to be the source coding counterpart of the latter.
]
T I. INTRODUCTION AND SYSTEMMODEL Notation: We write [a,b]=[a,a+1,...,b] for integers b>
I a; [a,b] = a if a = b; and [a,b] is empty otherwise. For
. Consider the problem of source coding with controllable
s a sequence of scalars x ,...,x , we write xn = [x ,...,x ]
c side informationillustrated inFig. 1.The encodercompresses 1 n 1 n
[ and x0 for the empty vector. The same notation is used for
1 asosuorcuercseymXbnol=. T[hXe1d,.e.c.,oXdenr], btoasaedmoensstahgeemWessoafgeRWb,itstakpeesr sequences of random variables Xn = [X1,...,Xn], or sets
Xn =[X ,...,X ].
v actions A for all i = 1,...,n, so as to control in a causal 1 n
i
6 fashion the measured side information sequence Yn. The A. System Model
2
action A is allowed to be a function of previously measured
9 i Thesystem,illustratedinFig.1,isdescribedbythefollow-
0 values Yi−1 of the side information, and the final estimate
ing random variables.
. Xˆ is obtained by the decoder based on message W and as a
1 caiusalfunctionon the side informationsamples. The problem • A source Xn with iBM of length L. The source Xn
0
consists of m blocks
3 ofcharacterizingthesetofachievabletuplesofrateR,average
1 distortion D and average action cost Γ was solved in [1, Sec. XL =(X ,...,X ) (1)
: II.E]undertheassumptionsofamemorylesssourceXnandof i (i−1)L+1 (i−1)L+L
v
i a memorylessprobabilisticmodelforthesideinformationYn withi∈[1,m],eachofLsymbols,sothatn=mL.The
X
when conditioned on the source and the action sequences1. alphabet is possibly changing across each L-block, that
r The distribution of the side information sequence given the is, we have X ∈ X , for L alphabets X ,...,X ,
a i t(i)+1 1 L
source and action sequencesis referredto as side information where we have defined
“vending machine” in [1].
t(i)=r(i−1,L), (2)
In this work, we generalize the characterizationof the rate-
distortion-cost performance for the set-up in Fig. 1, from
with r(x,y) being the remainder of x divided by y.
the memoryless scenario treated in [1], to a model in which • AmessageW ∈[1,2nR]withRbeingtheratemeasured
sourceand side information“vendingmachine”have in-block
in bits per source symbol.
memory(iBM).WithiBM,theprobabilisticmodelsforsource • An action sequence An with Ai ∈ At(i)+1 for L
and“vendingmachine”havememorylimitedtoblocksofsize alphabets A ,...,A .
1 L
L samples, where L does not grow with the coding length • A side information sequence Yn with Yi ∈Yt(i)+1 for
L alphabets Y ,...,Y .
1 L
in1[T3,heSemc.enIIti]onwehdicchhaisrarcetestrriizcatteidontoina[m1,odSeelc.wIiIth.E]acgtieonne-rianldizeepsentdheentressiudlet • AsourceestimateXˆnwithXˆi ∈Xˆt(i)+1forLalphabets
information. Xˆ1,...,XˆL.
y2a=30 xˆ2 y3xˆ3=0
y1a=20 xˆ1 y2=0 y3xˆ3=1
a1 y2a=31 y1=0 y2xˆ=21 yy3xxˆˆ=33=10
3
a xˆ
y =30 xˆ y 3=0
2 2 3
ya=21 xˆ y2=0 y3xˆ3=1
1 y2a=31 y1=11 y xˆ=21 y3xxˆˆ3=0
i=1 i=2 i=3 2 y =31
3
i=1 i=2 i=3
Fig. 2. An action codetree vn(w,·) for a given message w ∈ [1,2nR]
(Yi={0,1},n=3). Fig. 3. A decoder codetree un(w,·) for a given message w ∈ [1,2nR]
(Yi={0,1},n=3).
In order to simplify the notation, in the following, we will
writeX todenoteX alsofori>L,andsimilarlyforthe for some functions g : Ai ×Z →Y , with i ∈ [1,L].
i t(i)+1 i i
alphabets Ai, Yi and Xˆi. The variable are related as follows. Note that, as a special case, if the functions gi do
• The source Xn has iBM of length L in the sense that it not depend on the actions, equations (3) and (5) imply
that the sequences Xn and Yn are L-block memoryless
is characterized as
in the sense that their joint distribution factorizes as
X =f (Z ), (3) m P(xL,yL).
i t(i)+1 ⌈i/L⌉ i=1 i i
• The decoder, based on the received message W along
for some functions fi : Z →Xi, with i ∈ [1,L], Qwiththe currentandpastsamplesofthe sideinformation
where Zi, with i∈[1,m], is a memoryless process with sequence, produces the estimated sequence Xˆn. Specif-
probability distribution P(z). Note that (3) is equivalent ically, at each symbol i ∈ [1,n], the estimate Xˆ is
i
to the condition that the distribution P(xn) factorizes as selected as
mi=1P(xLi ). Xˆi =ui(W,Yi) (6)
• The encoder maps the source Xn into a message W ∈
[Q1,2nR] according to some function h : Xn → [1,2nR] for some functions ui :[1,2nR]×Yi →Xˆi. This condi-
as W = h(Xn). To denote functional, rather than more tional functional dependence is denoted as 1(xˆi|ui,yi),
generalprobabilistic,conditionaldependence,weuse the where un(w,·) represents the decoder codetree (or de-
notation 1(W|Xn). coder strategy) for a given message w ∈ [1,2nR] in
• The decoder observes the message W and takes actions the time interval i ∈ [1,n], that is, the collection of
An based also on the observation of the past samples functions ui(w,·) in (6) for all i ∈ [1,n]. A codetree
of the side information sequence. Specifically, for each un(w,·)(alongwiththesubtreesui(w,·)withi∈[1,n])
symbol i∈[1,n] the action Ai is selected as is illustrated in Fig. 3 for Yi ={0,1} and n=3.
Overall,theprobabilitydistributionoftherandomvariables
Ai =vi(W,Yi−1), (4) (Xn,Vn,An, Un,Yn,Xˆn) factorizes as
forsomefunctionsv :[1,2nR]×Yi−1 →A .Thiscondi- m
tionalfunctionaldepiendenceisdenotedas1(iai|vi,yi−1), " P(xLi )#P(vn,un|xn)1(an||vn,0yn−1) (7)
where vn=vn(w,·) represents the action codetree (or iY=1
m
action strategy) for a given message w ∈[1,2nR] in the
·1(xˆn||un,yn) P(yL||aL|xL) ,
timeintervali∈[1,n],thatis,thecollectionoffunctions i i i
" #
i=1
v (w,·) in (4) for all i ∈ [1,n]. A codetree vn(w,·) is Y
i
where we have used the directed conditioningnotation in [4].
illustrated in Fig. 2 for Y = {0,1} and n = 3. Note
i
Accordingly, we have defined
that the subtrees vi(w,·) with any i∈[1,n] can also be
obtained from Fig. 2.
L
P(yL||aL|xL)= P(y |ai,xL) (8)
• The side information has iBM of length L in the sense i
that it is generated as a function of the previous actions iY=1
taken in the same block and of the variable Z (cf. and similarly for the deterministic conditional relationships
⌈i/L⌉
(3)) as follows n
1(an||vn,0yn−1)= 1(a |vi,yi−1) (9)
i
Y =g (A ,...,A ,Z ), (5)
i t(i)+1 i−t(i) i ⌈i/L⌉ i=1
Y
xˆ
Z1 Z2 xˆ ,a y 3=0
2 3 3
X1 X2 X3 X4 y2=0 yxˆ3=1
xˆ,a 3
1 2
y =0 xˆ3
W 1 yxˆ2,=a31 y3xˆ=0
2 3
a y =1
1 3
A1 A2 A3 A4 xˆ ,a yxˆ3=0
2 3 3
y =0 xˆ
2 y 3=1
xˆ,a 3
1 2
Y1 Y Y3 Y4 y1=1 xˆ ,a yxˆ3=0
2 y2=31 3xˆ
2 3
y =1
3
i=1 i=2 i=3 i=4
Xˆ1 Xˆ2 Xˆ3 Xˆ4
Fig. 5. A codetree jn+1(w,·) for a given message w ∈ [1,2nR] (Yi =
{0,1},n=3).
Fig. 4. FDG fora source coding problem with iBM of length L=2 and
n=4sourcesymbols(andhencem=2blocks).Thetwoblocksareshaded
andthefunctional dependence onthesideinformation isdrawnwithdashed
lines. II. MAIN RESULTS
In this section, the rate-distortion-cost function R(D,Γ) is
derived and some of its properties are discussed. The next
and
n section illustrates various special cases and connections to
1(xˆn||un,yn)= 1(xˆi|ui,yi). (10) previous works.
i=1
Y
Afunctiondependencegraph(FDG)(see,e.g.,[4])illustrating A. Equivalent Formulation
thejointdistribution(7)forL=2andn=2(andthusm=2) We start by showing that the problem can be formulated
is shown in Fig. 4. in terms of a single codetree. This contrasts with the more
Remark 1. In (7), functions 1(an||vn,0yn−1) and naturaldefinitionsgivenin the previoussection, in which two
1(xˆn||un,yn) are fixed as they represent the map from separate codetrees, namely vn(w,·) and un(w,·), were used
the branches of the codetrees vn and un as indexed by the (see Fig. 2 and Fig. 3). Towards this end, we define a “joint”
side information sequence to the action a and estimate xˆ as codetree jn+1(w,·) = (j1(w,·),...,jn+1(w,·)) that satisfies
i i
illustrated in Fig. 2 and Fig. 3, respectively. the functional dependencies
Fixaanon-negativeandboundedfunctiondL(xL,xˆL)with 1(a |ji,yi−1)=1(a |vi,yi−1), (13)
domainXL×XˆLtobethedistortionmetricandanon-negative i i
and bounded function γL(aL,xL) with domain AL × XˆL and
to be the action cost metric. Under the selected metrics, a 1(xˆ |ji+1,yi)=1(xˆ |ui,yi) (14)
i i
triple (R,D,Γ)is saidto be achievablewith distortionD and
cost constraint Γ, if, for all sufficiently large m, there exist for all i∈[1,n]. The codetree jn+1(w,·) is illustrated in Fig.
codetrees such that 5 for n = 3. Note that the subtree j1(w,·) only specifies the
1 m action a1 to be taken at time i = 1, while the the leaves of
E[dL(XL,XˆL)]≤D+ǫ (11) the tree jn+1(w,·) are indexed solely by the estimated value
mL i i
Xi=1 xˆn.
Withthisdefinition,from(7),theprobabilitydistributionof
and
m the random variables (Xn,Jn+1, An,Yn,Xˆn) factorizes as
1
E[γL(AL,XL)]≤Γ+ǫ (12)
mL i i m
Xi=1 P(xL) P(jn+1|xn)1(an||jn,0yn−1) (15)
i
for any ǫ > 0. The rate-distortion-cost function R(D,Γ) is " #
i=1
Y
the infimumof all achievablerateswith distortionD and cost m
constraint Γ. ·1(xˆn||jn+1,yn) P(yL||aL|xL) ,
2 i i i
" #
Remark 2. The system model under study reduces to that Yi=1
investigated in [1, Sec. II.E] for the special case with memo- where we recall that we have 1(xˆn||jn+1,yn) =
2
ryless sources, i.e., with L=1. n 1(xˆ |ji+1,yi).
i=1 i
Q
B. Rate-Distortion-Cost Function where the joint distribution of the variables XL,YL,AL,XˆL
and of the codetrees VL and UL factorizes as
Using the representation in terms of a single codetree
given above, we now provide a characterization of the rate- P(xL)P(vL,uL|xL)1(aL||vL,0yL−1) (21)
distortion-cost function.
·1(xˆL||uL,yL)P(yL||aL|xL),
Proposition 1. The rate-distortion-cost function is given by
and the minimization is performed over the conditional dis-
1 tribution P(vL,uL|xL) of the codetreesunderthe constraints
R(D,Γ)= minI(XL;JL+1) (16)
L (18) and (19).
where the joint distribution of the variables XL,YL,AL,XˆL Remark 4. The rate-distortion-cost function in Proposition 1
and of the codetree JL+1 factorizes as doesnotincludeauxiliaryrandomvariables,sincethecodetree
JL+1 is part of the problem specification. This is unlike
P(xL)P(jL+1|xL)1(aL||jL,0yL−1) (17) the characterization given in [1] for the memoryless case.
·1(xˆL||jL+1,yL)P(yL||aL|xL), Moreover,problem(16)isconvexintheunknownP(jL+1|xL)
2
and hence can be solved using standard algorithms. It is also
andthe minimizationisperformedoverthe conditionaldistri- noted that, extending [5], one may devise a Blahut-Arimoto-
bution P(jL+1|xL) of the codetree under the constraints type algorithm for the calculation of the rate-distortion-cost
function. This aspect is not further investigated here.
1
E[dL(XL,XˆL)]≤D (18) Based on the definition of JL+1, we have the following
L
cardinalityboundonthenumberofcodetreestobeconsidered
and in the optimization (16):
1
E[γL(AL,XL)]≤Γ. (19) L−1
L |JL+1|≤|A ||Xˆ ||YL| (|Xˆ||A |)|Yi|. (22)
1 L i i+1
Proof: The achievability of Proposition 1 follows from
i=1
Y
classical random coding arguments. Specifically, the encoder
The following lemma shows that the this cardinality bound
drawsthecodetreesjn+1(w,·)forallw ∈[1,2n(R(D)+δ)]with
can be improved.
some δ >0, as follows. First, for each w ∈[1,2n(R(D)+δ)] a
concatenationofm codetreesjL+1(w,·) oflengthL+1,with Corollary1. Intheoptimization(16),thenumberofcodetrees
i
i ∈ [1,m], is generated, such that the constituent codetrees JL+1 can be limited as
jL+1(w,·) are i.i.d. and distributed with probability P(jL+1).
i |JL+1|≤|XL|+3 (23)
The codetree jn+1(w,·) is then obtained by combining the
leaves and the root of successive constituent codetrees: the without loss of optimality.
leaves of the past codetree specify the estimates for the
Proof: See Appendix B.
previous time instant, while the root of the next codetree
specify the action for the current time instant. The procedure Remark 5. The achievable scheme used to prove Proposition
is illustrated in Fig. 6. 1 adapts the actions only to the side information samples
correspondingtothesameL-block.Moreprecisely,theaction
Encoding is performed by looking for a message
w ∈ [1,2n(R(D)+δ)] such that the corresponding pair Ai depends, through the selected codetree, only on the side
(xn,jn+1(w,·)) is (strongly) jointly typical with respect to information samples Yi−t(i),...,Yi−1. Since the problem defi-
the joint distribution P(xL)P(jL+1|xL), when the sequences nition allows, via (4), for actions that depend on all past side
(xn,jn+1(w,·)) are seen as the memoryless m-sequences information samples, namely Yi−1, this result demonstrates
(xL,jL+1(w,·)),...,(xL,jL+1(w,·)). By the covering lemma thatadaptingtheactionsacrosstheblockscannotimprovethe
1 1 m m
[6, Lemma 3.3], rate 1/L·I(XL;JL+1) suffices to guaran- rate-distortion-costfunction.Thisisconsistentwiththefinding
in [1], where it is shown that adaptive actionsdo notimprove
tee the reliability of this step. Moreover, if the distribution
P(jL+1|xL)isselectedsoastosatisfy(18) and(19),then,by the rate-distortion performance for a memoryless model, i.e.,
with L = 1. Similarly, one can conclude from Proposition 1
the typical average lemma [6], the constraints (11) and (12)
that, while adapting the estimate Xˆ to the side information
are also guaranteed to be met for sufficiently large n. The i
samples within the same L-block, namely Y ,...,Y , is
proof of the converse can be found in Appendix A. i−t(i) i
generally advantageous, adaptation across the blocks is not.
Remark 3. The rate-distortion-cost function can also be ex-
This extends the results in [3], in which it is shown that, for
pressedintermsoftwoseparatecodetreesusingthedefinitions
L = 1, the estimate can depend only on the current value of
given in Sec. I-A. Specifically, following similar steps as in
the side information without loss of optimality.
theproofofProposition1,therate-distortion-costfunctioncan
be expressed as the minimization III. SPECIAL CASES AND EXAMPLES
1 In this section, we detail some further consequences of
R(D,Γ)= minI(XL;VL,UL) (20) Proposition 1 and connections with previous work.
L
xˆ2 xˆ E] can be seen to be a special case of Proposition 1.
4
y2 =0 y =0 B. Action-IndependentSide Information
4
xˆ ,a xˆ ,a Here we consider the case in which the side information
y11=02 y2xˆ2=1 y33=04 yxˆ4=1 inPso(tyaLcnte|ixoeLdn).tionUdbneedpeeirnndcthelunisdt,eadsthsiuantmtpihst,eionwm,eotdhheeal,vaecatniPodn,(yfsrLeoq|m|uaeLn(|2cx0eL),)dothe=es
a a 4 rate-distortion function is given by
1 3
1
R(D)= minI(XL;UL), (24)
L
xˆ xˆ
y2 =20 y4 =40 owfhtehree cthoedejtorienetUdiLstrfiabcuttoiorinzeosfatshe variables XL,YL,XˆL and
xˆ ,a xˆ ,a
y1=12 y3=14 P(xL)P(uL|xL)1(xˆL||uL,yL)P(yL|xL), (25)
1 3
xˆ xˆ and the minimizationis performedoverthe conditionaldistri-
y2 2=1 y 4=1 bution P(uL|xL) of the codetrees under the constraint (18).
4
Note that, given the absence of actions, we have used the
P(jL+1) P(jL+1) formulation in terms of individual codetrees discussed in
Remark 3 in order to simplify the notation. Using arguments
similar to Corollary 1, one can show that the size of the
codetree alphabet can be limited to |UL|≤|XL|+2 without
lossofoptimality.ForL=1,thecharacterization(24)reduces
xˆ ,a xˆ to the one derived in [3, Sec. II].
2 3 4
y =0 y =0 C. Block-Feedforward Model
2 4
xˆ ,a xˆ ,a Asaspecificinstanceofthesettingwithaction-independent
1 2 3 4
y =0 y =0 side information, we consider here the block-feedforward
1 3 modelinwhichwehaveY =X forallinotmultipleofL
xˆ ,a xˆ i i−1
y2 =31 y 4=1 andYi equalto a fixedsymbolin Yi otherwise.Thismodelis
a 2 4 related to the feedforward set-up studied in [7], [8], [9] with
1
thedifferencethatherefeedforwardislimitedtowithintheL-
blocks.Inotherwords,thesideinformationisY =X only
i i−1
xˆ ,a xˆ
2 3 4 if Xi−1 is in the same L-block as Yi and is not informative
y =0 y =0 otherwise.Wenowshowthat,similarto[8],therate-distortion
2 4
xˆ ,a xˆ ,a functionwith block-feedforwardcan be expressed in terms of
1 2 3 4
y =1 y =1 directed information and does not entail an optimization over
1 3
the codetrees.
xˆ ,a xˆ
2 3 4
y =1 y =1 Corollary 2. For the block-feedforward model, the rate-
2 4
distortion function is given by
i=1 i=2 i=3 i=4 i=5 1
R(D)= minI(XˆL →XL) (26)
L
Fig.6. IllustrationoftheachievableschemeusedintheproofofProposition where the joint distribution of the variables XL, YL and XˆL
1forbinaryalphabetsYi={0,1}withm=2andL=2.Inthetopfigure,
thecodetreesj3i(w,·)fori=1,2,whicharegeneratedi.i.d.withprobability factorizes as
P(jL+1), are depicted. Inthe bottom figure, the resulting codetree j5(w,·)
isshown.Itisnotedthattheactiona3 inthecodetreej5(w,·)inthebottom P(xL)P(xˆL|xL)P(yL|xL), (27)
figure is obtained from the codetree j3i(w,·) with i = 2 on the top, and is
thusindependent ofthevalue ofy2. andthe minimizationis performedoverthe conditionaldistri-
bution P(xˆL|xL) under the constraint (18).
Remark 6. In the feedforward model studied in [7], [8], [9],
A. Memoryless Source (L=1) feedforward of the source Xn is not restricted to take place
As mentioned in Remark 2, if L = 1, the model at only within the L-blocks, namely we have Yi =Xi−1 for all
hand reduces to the standard one with memoryless sources, i∈[1,n]. As a result, the rate-distortionfunctionis provedin
in which the joint distribution of Xn and Yn factorizes as [8], [9] to be given by the limit of (26) over L.
n P(x ,y ).Thismodelwasstudiedin[1],wheretherate- Proof: The achievability is obtained by using concate-
i=1 i i
distortion-cost function was derived. The result in [1, Sec. II- natedcodetreesoflengthLsimilartoProposition1.However,
Q
unlike Proposition1, the codetreesare generatedaccordingto Z Z
1 2
athcehideivsatrbiibluittyioinspc(oxˆmLp||l0etxeLd−a1s)ians[d8o]n,e[9in].[A8]s,f[o9r].tThehecopnrovoefrsoef, X1 X2
starting from (24), we write
W
L
I(XL;UL) = I(X ;UL|Xi−1)
i
Xi=1 A1 A2
L
= I(X ;UL,Xˆi|Xi−1)
i
i=1
X
L
Y Y Y
≥ I(Xi;Xˆi|Xi−1) 11 Y12 21 22
i=1
X
= I(XˆL →XL), (28)
wherethesecondequalityfollowssinceXˆiisafunctionofthe Xˆ1 Xˆ2
codetreeUL andofYi =Xi−1;theinequalityfollowsbythe
non-negativityofthemutualinformation;andthelastequality Fig.7. FDGforthemodelwithsideinformation repeat request.
is a consequenceof the definition of directed information[4].
assume that the channel P(y |x) for the first measurement
1
Example 1. Consider a binary source with iBM of length
is an erasure channel with erasure probability ǫ. Moreover,
L = 2 and block-feedforward such that variables X , for all
i the channel P(y |x,a) for the second measurement is an
2
oddi,arei.i.d.Bern(p), with0≤p≤0.5,while foralleven
independent and identical erasure channel if a = 1, while
i we have X = X ⊕Q with Q being i.i.d. Bern(q),
i i−1 i i it produces Y equal to the erasure symbol with probability
2
with 0 ≤ q ≤ 0.5 and independent of X for all odd i.
i 1 if a = 0. In other words, the action a = 1 corresponds
Assuming Hamming distortion d2(x2,xˆ2) = 2 1(x ,xˆ ),
i=1 i i to performing a second measurement of the side information
from Corollary2, we easily obtain that, if D <(p+q)/2,the
over an independentrealization of the same erasure channel.
P
rate-distortion function is given as
It is apparent that, if Y = X, one can set A = 0 without
1
1 loss of optimality. Instead, if Y equals the erasure symbol,
min [H (p)−H (D )+H (q)−H (D )] (29) 1
D1+D2≤2D 2 2 2 1 2 2 2 then, in the absence of action cost constraints, it is clearly
optimaltosetA=1.Insodoing,thesideinformationchannel
where the minimization is under the constraints D ≤ p and
1 is converted into an equivalent erasure channel with erasure
D ≤q, and is zero otherwise.
2 probability ǫ2. Therefore, the rate-distortion is given by [10],
D. Side Information Repeat Request [11]
D
Consider the situation in which the decoder at any time i, R(D,Γ)=ǫ2 1−H (30)
2 ǫ2
upon the observation of the side information Yi, can decide (cid:18) (cid:18) (cid:19)(cid:19)
whethertotakeasecondmeasurementofthesideinformation, for D ≤ ǫ2/2 and zero otherwise, as long as the action
thuspayingtheassociatedcost,ornot.Toelaborate,assumea
cost budget Γ is large enough. More specifically, given the
memorylesssourceXn with distributionP(x). Atanytime i,
discussion above,itcan be seen thatΓ≥ǫ sufficesto achieve
the first observation Y of the side information is distributed
i1 (30).
accordingto the memorylesschannelP(y |x) when the input
1
is X = x, while the second observation Y depends on the
i i2 IV. CONCLUDING REMARKS
action A = a via the memoryless channel P(y |x,a) with
i 2
input X =x. Models with in-block memory (iBM), first proposed in the
i
This scenario can be easily seen to be a special case of the contextofchannelcodingproblemsin [2]andhereforsource
modelunderstudywithiBMofsizeL=2.Thecorresponding coding, provide tractable extensions of standard memoryless
FDG is illustrated in Fig. 7. By comparingthis FDG with the models. Specifically, in this paper, we have presented results
general FDG in Fig. 4, it is seen that the model under study for a point-to-pointsystem with controllable side information
in this section can be obtainedfromthe one presentedin Sec. atthereceiverandiBM.Interestinggeneralizationsincludethe
I-A by appropriately setting the alphabets of given subset of investigation of multi-terminal models.
variables to empty sets and by relabeling.
A characterization of the rate-distortion-cost function can ACKNOWLEDGMENT
be easily derived as a special case of Proposition 1. Here TheauthorwouldliketothankGerhardKramerfortheveryuseful
we focus on a specific simple example. In particular, we comments and suggestions.
APPENDIX A Now, fix the so obtained distribution P(xL|jL+1) and recall
PROOF OF THECONVERSE OF PROPOSITION 1 that the other terms in (32) are also fixed by the problemdef-
inition. Now, the quantities appearingin Proposition 1 can be
For any code achieving rate R with distortion D and cost
writtenasconvexcombinationsoffunctionsofthetermsfixed
Γ, we have the following series of inequalities:
above, in which the distribution P(jL+1) defines the coeffi-
nR ≥ H(W)=I(W;Xn) cientsofthecombinations.Specifically,wehave:(i)thedistri-
(=a) m H(XiL)− m H(XiL|X(i−1)L,W) (bbuutitononPe)(,xwL)hi=ch fijxLe+s1PH((jXL+L1));P((iix)Lt|hjeL+c1o)nfdoitrioanllaxlLen∈troXpLy
P
i=1 i=1 H(XL|JL+1) = P(jL+1)H(XL|JL+1 = jL+1); and
X X jL+1
m m
(=b) H(XL)− H(XL|X(i−1)L,W,Y(i−1)L) (iii) the averagesPE[dL(XL,XˆL)] and E[γL(AL,XL)]. It
i i follows by the Caratheodory theorem that we can limit the
i=1 i=1
X X alphabet of JL+1 as in (23) without loss of optimality.
m m
(=c) H(XL)− H(XL|X(i−1)L,W,Y(i−1)L,J¯L+1)
i i i REFERENCES
i=1 i=1
X X
m m [1] H.PermuterandT.Weissman,“Sourcecodingwithasideinformation
(d)
≥ H(XL)− H(XL|J¯L+1) “Vending Machine”,” IEEE Trans. Inform. Theory, vol. 57, no. 7, pp.
i i i 4530-4544, Jul.2011.
Xi=1 Xi=1 [2] G. Kramer, “Information networks in-block memory,” in Proc. IEEE
(=e) mH(XL)−mH(XL|J¯L+1,T) InformationTheoryWorkshop(ITW2012),Lausanne,Switzerland,Sep.
i 2012.
≥ mI(XL;J¯L+1), [3] T.WeissmanandA.ElGamal,“Sourcecodingwithlimited-look-ahead
side information atthe decoder,” IEEETrans.Inform. Theory, vol.52,
where(a)followsduetothe blockmemoryofthesourceXn; no.12,pp.5218-5239, Dec.2006.
[4] G. Kramer, Directed Information for Channels with Feedback, volume
(b) follows due to the Markov chain XL−(X(i−1)L,W)−
i ETH Series in Information Processing. Vol. 11, Konstanz, Germany:
Y(i−1)L; (c) is obtained by defining J¯L+1 as the subtree of Hartung-Gorre Verlag,1998.
i
Ji+1 corresponding to Y(i−1)L, respectively, and noting that [5] Fr.Dupuis,W.Yu,andF.M.J.Willems,“Blahut-Arimotoalgorithmsfor
J¯L+1 is a function of (W,Y(i−1)L); (d) is due to the fact computingchannelcapacityandratedistortionwithsideinformation,”in
i Proc.IEEEInt.Sym.Inform.Theory(ISIT2004),Chicago,IL,Jun./Jul.
that conditioning cannot increase entropy; (e) is obtained by 2004.
defining a random variable T uniformly distributed in the set [6] A. ElGamaland Y.-H. Kim,Network Information Theory, Cambridge
University Press,2012.
[1,m] and independent of all other variables, and also the
[7] T.WeissmanandN.Merhav,“Oncompetitivepredictionanditsrelation
variables J¯L+1 = J¯L+1 and XL = XL, and using the fact torate-distortion theory,” IEEETrans.Inform. Theory, vol. 49,no.12,
T T
that the distribution of XL does not depend on i. pp.3185-3194,Dec.2003.
i [8] R. Venkataramanan and S. S. Pradhan, “Source coding with feed-
Giventhedefinitionsabove,andsettingAL =AL,thejoint
T forward: Rate-distortion theorems and error exponents for a general
distribution of the random variables at hand factorizes as source,” IEEE Trans. Inform. Theory, vol. 53, no. 6, pp. 2154-2179,
Jun.2007.
P(xL)P(¯jL+1|xL)1(aL||¯jL,0yL−1) (31) [9] I.Naiss andH.Permuter, “Computable boundsforratedistortion with
feed-forward forstationary andergodic sources,”arXiv:1106.0895v1.
·1(xˆL||¯jL+1,yL)P(yL||aL|xL), [10] S.Diggavi,E.Perronand.ETelatar,“LossysourcecodingwithGaussian
2
or erased side-information,” in Proc. IEEE Int. Symp. Inform. Theory
where we have defined P(¯jL+1|xL) = (ISIT2009),pp.1035-1039, 2009.
1 m P(¯jL+1|xL,t). Note that, in showing (31), it is [11] S. Verdu´ and T. Weissman, “The information lost in erasures,” IEEE
m t=1 Trans.Inform.Theory,vol.54,no.11,pp.5030-5058,Nov.2008.
critical that, as per (5), the side information YL in the ith
P i
block dependsonly on the actions in the ith block. The proof
is concluded by noting that the defined random variables
also satisfy the constraints (18) and (19) due to the fact that
any code at hand must satisfy the conditions (11) and (12),
respectively.
APPENDIX B
PROOF OF COROLLARY 1
Assume that a rate is achievable for some distribution
P(jL+1|xL), wherethecardinalityof JL+1 islimited onlyby
the count of available codetrees as in (22). We want to show
that the same rate can be achievedby limiting the alphabetof
available codetrees as in (23). To this end, we first write the
joint distribution (17) as
P(jL+1)P(xL|jL+1)1(aL||jL,0yL−1) (32)
·1(xˆL||jL+1,yL)P(yL||aL|xL).
2