Blahut-Arimoto Algorithm and Code Design for Action-Dependent Source Coding Problems

0.53 MB·English

by  Kasper Flœ Trillingsgaard

1
Blahut-Arimoto Algorithm and Code Design
for Action-Dependent Source Coding Problems
Kasper Fløe Trillingsgaard, Osvaldo Simeone, Petar Popovski and Torben Larsen
3
1
0
2
Abstract

n
a
J
The source coding problem with action-dependent side information at the decoder has recently
5
2
been introduced to model data acquisition in resource-constrained systems. In this paper, an efficient

]
T algorithm for numerical computation of the rate-distortion-cost function for this problem is proposed,
I
s. and a convergence proof is provided. Moreover, a two-stage code design based on multiplexing is put
c
[ forth, whereby the first stage encodes the actions and the second stage is composed of an array of

1
classical Wyner-Ziv codes, one for each action. Specific coding/decoding strategies are designed based
v
0
9 onLDGMcodesandmessagepassing.Throughnumericalexamples,theproposedcodedesignisshown
1
6 to achieve performance close to the lower bound dictated by the rate-distortion-cost function.
.
1
0
3
Index Terms
1
:
v
Xi Rate-distortiontheory,sideinformation“vendingmachine”,Blahut-Arimotoalgorithm,codedesign,
r
a LDGM, message passing.
I. INTRODUCTION
The source coding problem in which the decoder can take actions that affect the availability or
quality of the side information at the decoder was introduced in [1]. The problem generalizes the
well-known Wyner-Ziv set-up and can be used to model data acquisition in resource-constrainted
systems, such as sensor networks. In the model studied in [1], each action is associated a cost
January29,2013 DRAFT
2
and the system design is subject to an average cost constraint. The information-theoretic analysis
of the problem was fully addressed in [1]. In this paper, instead, we tackle the practical open
issues, namely the computation of the rate-distortion-cost function and code design.
Specifically,therate-distortion-costfunctionforthesourcecodingproblemwithaction-dependent
side information was derived in [1]. However, no specific algorithm was proposed for its com-
putation. A first contribution of this paper is to propose such an algorithm by generalizing the
classical Blahut-Arimoto (BA) approach, which was introduced for the Wyner-Ziv problem in
[2]. Convergence of the algorithm is also proved.
Moreover, while the theory in [1] demonstrates the existence of coding and decoding strategies
able to achieve the rate-distortion-cost bound, practical code constructions have not been inves-
tigated yet. It is recalled that, for classical lossy source coding problems, codes that have been
able to achieve rate-distortion bound include Low Density Generator Matrix (LDGM) codes [3],
polar codes [4] and trellis-based quantization codes [5]. For the Wyner-Ziv problem, efficient
codes include compound LDPC/LDGM codes [6] and polar codes [4]. A second contribution of
this paper is hence the study of code design for source coding problems with action-dependent
side information. As shown in [1], optimal codes for this problem have a successive refinement
structure, in which the first layer produces the action sequence and the refinement layer uses
binning to leverage the side information at the decoder. Here, we first observe that a layered
code structure in which the refinement layer uses a multiplexing of separate classical Wyner-Ziv
codes, one for each action, is optimal. This allows us to simplify the code structure with respect
to the successive refinement strategy in [1]. LDGM-based codes with message passing encoding
are designed and demonstrated via numerical results to perform close to the rate-distortion-cost
function.
The paper is organized as follows. In Section II, the action-dependent source coding problem is
describedandresultsfrom[1]aresummarized.InSectionIII,wedescribetheproposedalgorithm
for computation of the rate-distortion-cost function, and in Section IV, a practical code design
DRAFT January29,2013
3
Fig. 1. Source coding with action-dependent side information.
is proposed. Finally, in Section V, we present numerical results for a specific example.
A. Notation
Throughout this work, we let upper case, lower case and calligraphic letters denote random
variables, values and alphabets of the random variables, respectively. For jointly distributed
random variables, P (x), P (x|y) and P (x,y) denote the probability mass function (pmf)
X X|Y X,Y
of X, the conditional pmf of X given Y and the joint pmf of X and Y. To simplify notation,
the subscripts of the pmfs may be omitted, e.g., P(x|y) may be used instead of P (x|y).
X|Y
The notation Xn represents the tuple (X ,X ,...,X ), and [a,b] where a,b ∈ Z with a < b
1 2 n
denotes the set of integers {a,a+1,...,b−1,b}. Moreover, Z = {0,1,...}, N = Z \{0}
+ +
and 1 denotes the indicator function, and is one when cond is true, and zero otherwise.
{cond}
The notation (cid:98)·(cid:99) and (cid:100)·(cid:101) denotes the floor and ceiling operators, respectively.
II. BACKGROUND
In this section, we recall the definition of source coding problems with action-dependent side
information and review the rate-distortion-cost function obtained in [1].
A. System Model
The source coding problem with action-dependent side information introduced in [1] is il-
lustrated in Fig. 1. In this problem, the source Xn ∈ Xn is memoryless and each sample is
distributed according to the pmf P . At the encoder, the encoding function
X
(cid:2) (cid:3)
f : Xn → 1,(cid:98)2nR(cid:99) , (1)
January29,2013 DRAFT
4
(cid:2) (cid:3)
maps the source Xn into a message M ∈ 1,(cid:98)2nR(cid:99) , where R denotes the rate in bits per sample.
At the decoder, an action sequence An ∈ An is chosen according to an action strategy
(cid:2) (cid:3)
g : 1,(cid:98)2nR(cid:99) → An, (2)
which maps the message M into an action sequence An. Based on An, the side information Yn ∈
Yn is conditionally independent and identically distributed (iid) according to the conditional pmf
P so that we have
Y|X,A
n
(cid:89)
P (yn|xn,an) = P (y |x ,a ). (3)
Yn|Xn,An Y|X,A i i i
i=1
The decoder makes a reconstruction Xˆn ∈ Xˆn of Xn according to the decoding function
h : (cid:2)1,(cid:98)2nR(cid:99)(cid:3)×Yn → Xˆn, (4)
which maps message M and side information Yn into the estimate Xˆn.
The action cost function ∆(a) : A → R is defined such that ∆(a) = 0 for some a ∈ A and
+
∆ = max ∆(a) < ∞, and the distortion function d(x,xˆ) : X ×Xˆ → R is defined such
max a∈A +
ˆ
that for each x ∈ X there is an xˆ ∈ X satisfying d(x,xˆ) = 0. The rate-distortion-cost tuple
(R,D,C) is then said to be achievable if and only if, for all ε > 0, there exist an encoding
function f, an action function g and a decoding function h, for all sufficiently large n ∈ N,
satisfying the distortion constraint
(cid:34) (cid:35)
n
(cid:88)
E d(X ,Xˆ ) ≤ n(D+ε) (5)
i i
i=1
and the action cost constraint
(cid:34) (cid:35)
n
(cid:88)
E ∆(A ) ≤ n(C +ε). (6)
i
i=1
The rate-distortion-cost function, denoted as R(D,C), is defined as the infimum of all rates R
such that the tuple (R,D,C) is achievable.
DRAFT January29,2013
5
Fig. 2. Optimal encoder for source coding problems with action-dependent side information.
B. Rate-Distortion-Cost Function
The rate-distortion-cost function R(D,C) was derived in [1] and is summarized below.
Lemma 1. ([1, Theorem 1]) The rate-distortion-cost function for the source coding problem
with action-dependent side information is given as
R(D,C) = minI(X;A)+I(X;U|Y,A), (7)
P (x,y,a,u) = P (x)P (u|x)1 P (y|x,a), (8)
X,Y,A,U X U|X {η(u)=a} Y|X,A
and the minimization is over all pmfs P and deterministic functions η : U → A under which
U|X
the conditions
E[d(X,Xˆopt(U,Y))] ≤ D, (9)
and
E[∆(A)] ≤ C (10)
hold. The function Xˆopt : U ×Y → Xˆ denotes the best estimate of X given U and Y, i.e.,
Xˆopt(u,y) = argminE[d(X,xˆ)|U = u,Y = y]. (11)
xˆ∈Xˆ
Moreover, the cardinality of the set U can be restricted as |U| ≤ |X||A|+2.
January29,2013 DRAFT
6
C. Optimal Coding Strategy
The proof of achievability of the rate-distortion-cost function in [1] shows that an optimal
encoder has the structure illustrated in Fig. 2 and consists of the following two steps.
• Action Coding: The source sequence Xn is mapped to an action sequence An. The ac-
tion sequence is selected from a codebook C of about 2nI(X;A) codewords, each type
A
approximately equal to P . The index Bk identifies the selected codeword An, and hence
A
consists of k, approximately equal to nI(X;A), bits. The selection of An is done with
the aim of ensuring that An and Xn are jointly typical with respect to the joint pmf
P (x,a) = P (a|x)P (x).
X,A A|X X
• Source Coding: Given the action sequence An, a source codebook is chosen out of a set of
around 2nI(X;A) codebooks, one for each codeword in C . Each codeword Un in the selected
A
source codebook has a joint type with An close to P , and the number of codewords is
A,U
about 2nI(X;U|A). The source sequence is mapped to a sequence Un taken from the selected
codebook with joint type P and with the objective of ensuring that Xn, An and Un are
A,U
jointly typical with respect to the joint pmf P (x,a,u). Each source codebook is divided
X,A,U
into around 2nI(X;U|A,Y) subcodebooks, or bins, in order to leverage the side information at
the receiver using Wyner-Ziv decoding.
The message M is given by the concatenation of the bits Bk and Bks and thus the overall rate
s
of the action code and the source codes is given by (7). Upon receiving the message M from the
encoder, the decoder first reconstructs the action sequence An. The action sequence is used to
measure the side information Yn. As An is known, the decoder also knows the source codebook
from which Un is selected, and Un is then recovered by using Wyner-Ziv decoding based on
the side information Yn. In the end, the final estimate Xˆn is obtained as Xˆ = Xˆopt(U ,Y ) for
i i i
i ∈ [1,n].
DRAFT January29,2013
7
III. COMPUTATION OF THE RATE-DISTORTION-COST FUNCTION
In this section, we first reformulate the problem in (7) by introducing Shannon strategies. This
result is then used to propose a BA-type algorithm for the computation of the rate-distortion-cost
function (7).
A. Shannon Strategies
We first observe that, from Lemma 1, it is sufficient to restrict the minimization to all joint
distributions for which A is a deterministic function A = η(U). Moreover, the final estimate of
ˆ
X in (11) is a function of both U and Y. Based on these facts, we define a Shannon strategy
T ∈ T ⊆ X|Y|×A as a vector of cardinality |Y|+1, in which the first |Y| elements are indexed
ˆ
by the elements in Y and T(y) ∈ X for y ∈ Y, and the last element is denoted a(T) ∈ A. We
also define the disjoint sets T a = {t ∈ T : a(t) = a} for all actions a ∈ A. The rate-distortion-
cost function (7) can be restated in terms of the defined Shannon strategies as formalized in the
next proposition.
Proposition 1. Let T ∈ T ⊆ X|Y|×A denote a Shannon strategy vector as defined above. The
rate-distortion-cost function in (7) can be expressed as
R(D,C) = minI(X;a(T))+I(X;T|Y,a(T)), (12)
where the joint pmf P is of the form
X,Y,T
P (x,y,t) = P (x)P (t|x)P (y|a(t),x), (13)
X,Y,T X T|X Y|A,X
and the minimization is over all pmfs P under the constraints
T|X
(cid:88)
E[∆(A)] = P (x)P (t|x)∆(a(t)) ≤ C (14)
X T|X
t∈T,x∈X
and
(cid:88)
E[d(X,T(Y))] = P (x,y,t)d(t(y),x) ≤ D. (15)
X,Y,T
t∈T,x∈X,y∈Y
January29,2013 DRAFT
8
Moreover, the cardinality of the alphabet T can be restricted as |T | ≤ |X||A|+2.
Proof: Given an alphabet U, a pmf P and a function η : U → A, the sum of the two
U|X
mutual informations in (7) can be seen to be equal to the sum of the two mutual informations in
(12) and the average distortion and cost in (9) and (10) to be equal to (15) and (14), respectively,
by defining P as follows. For each u ∈ U, define a strategy t with P (t|x) = P (u|x)
T|X T|X U|X
such that a(t) = η(u) and t(y) = Xˆopt(u,y) for y ∈ Y.
Remark. The characterization in Proposition 1 generalizes the formulation of the Wyner-Ziv
rate-distortion function in terms of Shannon strategies given in [2].
The following lemma extends to the rate-distortion-cost function R(D,C) some well-known
properties for the rate-distortion function (see, e.g. [7], [8]). This will be useful in the next
section when discussing the computation of R(D,C).
Lemma 2. The following properties hold for the rate distortion cost-function R(D,C):
1) R(D,C) is non-increasing, convex and continuous for D ∈ [0,∞) and C ∈ [0,∞).
2) R(D,C) is strictly decreasing in D ∈ [0,D (C)] and R(D (C),C) = 0, where
max max
(cid:88)
D (C) = min P (x,y,t)d(t(y),x), (16)
max X,Y,T
PT
t∈T,x∈X,y∈Y
under the constraint
(cid:88)
E[∆(a(T))] = ∆(a(t))P (t) ≤ C. (17)
T
t∈T
3) For all D ∈ [0,D (C)], the minimum in (12) is attained when the distortion inequality
max
(15) is satisfied with equality.
Proof: The lemma is proved by the arguments in [8, Lemma 10.4.1].
B. Computation of the Rate-Distortion-Cost Function
DRAFT January29,2013
9
Algorithm 1 BA-type Algorithm for Computation of the Rate-Distortion-Cost Function
input: Lagrange multipliers s ≤ 0 and m ≤ 0.
output: R(D ,C ) with C and D as in (19)-(20).
s,m s,m s,m s,m
initialize: P
T|X
repeat
Compute Q as in (25).
A
Compute Q as in (26).
T,Y
Minimize F(P ,Q ,Q ) with respect to P using Algorithm 2.
T|X T,Y A T|X
until convergence
P∗ ← P
T|X T|X
In order to derive a BA-type algorithm to solve the problem in (12), we introduce Lagrange
multipliers m for the cost constraint in (14) and s for the distortion constraint (15). The following
proposition provides a parametric characterization of the rate-distortion-cost function in terms
of the pair (s,m).
Proposition2. Foreachs ≤ 0andm ≤ 0,definetherate-distortion-costtuple(R ,D ,C )
s,m s,m s,m
via the following equations
R = sD +mC
s,m s,m s,m
+min{I(X;A)+I(X;T|Y,a(T))−sE[d(X,T(Y))]−mE[∆(a(T))]}, (18)
P
T|X
(cid:88)
C = P (x)P∗ (t|x)∆(a(t)), (19)
s,m X T|X
t∈T,x∈X
(cid:88)
D = P (x)P∗ (t|x)P (y|x,a)d(t(y),x), (20)
s,m X T|X Y|X,A
t∈T,x∈X,y∈Y
where P∗ denotes a minimizing pmf P for the optimization problem in (18). Then, the
T|X T|X
following facts hold
January29,2013 DRAFT
10
1) The tuple (R ,D ,C ) lies on the rate-distortion-cost function, i.e.,
s,m s,m s,m
R = R(D ,C ). (21)
s,m s,m s,m
2) Every point (R,D,C) on the rate-distortion-cost function for D ∈ [0,D (C)] can be
max
written as (18)-(20) for s ≤ 0 and m ≤ 0;
3) The rate-distortion-cost function is given as
R(D,C) = max(R +s(D−D )+m(C −C )). (22)
s,m s,m s,m
s≤0
m≤0
Proof: The proposition above follows by strong duality as guaranteed by Slater’s condition
[9, Section 5.2.3], and can also be derived directly as in [7].
Given the proposition above, one can trace the rate-distortion-cost function by solving problem
(18) and using (19) and (20) for all s ≤ 0 and m ≤ 0. Inspired by the standard BA approach,
we now show that problem (18) can be solved by using alternate optimization with respect to
P and appropriately defined auxiliary pmfs Q and Q . To do this, we define the function
T|X T,Y A
F(·) of P and auxiliary pmfs Q and Q as in (23),
T|X T,Y A
(cid:88)
F(P ,Q ,Q ) = D (P ||Q )− P (x,y,t)logP (y|x,a(t))
T|X T,Y A KL Y,A A X,Y,T Y|X,A
x∈X,y∈Y,t∈T
(cid:88) (cid:88)
+ P (x)D (P (·,·|x)||Q )−s P (x,y,t)d(t(y),x)
X KL Y,T|X T,Y X,Y,T
x∈X t∈T,x∈X,y∈Y
(cid:88)
−m ∆(a(t))P (x)P (t|x), (23)
X T|X
t∈T,x∈X
where D (P||Q) denotes the Kullback-Leibler (KL) divergence1 and P , P and P
KL X,Y,T Y,T|X Y,A
are calculated from the joint pmf (13). We then have the following result.
Proposition 3. For any s ≤ 0 and m ≤ 0, we have
R(D ,C ) = sD +mC + min F(P ,Q ,Q ), (24)
s,m s,m s,m s,m T|X T,Y A
PT|X,QT,Y,QA
1The Kullback-Leibler divergence [8] is defined as D (P||Q)=(cid:80) P(i)log P(i) for pmfs P and Q.
KL i 2 Q(i)
DRAFT January29,2013