On the Many-Help-One Problem with Independently Degraded Helpers

1.1 MB·

by  Albrecht Wolf

1
On the Many-Help-One Problem with
Independently Degraded Helpers
Albrecht Wolf, Diana Cristina Gonza´lez, Meik Do¨rpinghaus, Member, IEEE,
Jose´ Caˆndido Silveira Santos Filho, Member, IEEE, and Gerhard Fettweis, Fellow, IEEE
Abstract
Thisworkprovidesnewresultsfortheseparatedencodingofcorrelateddiscretememorylesssources.Weaddressthescenario
where an arbitrary number of auxiliary sources (a.k.a. helpers) are supposed to help decode a single primary source errorless.
7 Therate-distortionanalysisofsuchasystemestablishestheso-calledmany-help-oneproblem.Wefocusonthecaseinwhichthe
1 auxiliary sources are conditionally independent, given the primary source. We provide an inner and an outer bound of the rate-
0 distortion region for what we call the strong many-help-one problem, where the auxiliary sources must be also recovered under
2 certaindistortion constraints. Furthermore, based on known resultsfrom theCEOproblem weprovide the admissiblerateregion
for what we call the weak many-help-one problem, where the auxiliary sources do not need to be recovered serving merely as
n
sideinformationtohelprecovertheprimarysource.Bothscenariosfindimportantapplicationtoemergingcooperativetechniques
a
in which adirect-link communication isassisted by multiple lossy relaying-link transmissions. Motivated by this application, we
J
specialize the referredscenarios tobinary sources and the Hamming distortion measure. Remarkably, we show that for theweak
3
many-help-one problem the independent decoding of the auxiliary sources can achieve optimal performance.
2

Index Terms
]
T
Admissable rate region, correlated sources, distributed source coding, many-help-one problem, rate-distortion region.
I
.
s
c I. INTRODUCTION
[
Multiterminalsource codinghas a rich history.Slepian and Wolf [1] were the first to characterise the problemof distributed

1 encodingof multiple correlatedsources. In their seminal paper,the admissible rate region for the lossless distributed encoding
v of two correlated sources was derived. A simple proof of the Slepian-Wolf result with extension to an arbitrary number of
6 correlated sources was presented by Cover [2]. Wyner [3] and Ahlswede and Ko¨rner [4] considered a different problem, in
1
whichan auxiliaryrandomvariable(i.e.,side information)was introducedto expandthe rate regionof a lossless single-source
4
coding problem. In that setup coded (or partial) side information was available at the decoder. Wyner and Ziv [5] presented a
6
0 generalizationtolossysingle-sourcecodingwithuncodedsideinformation.Thiswasthefirstcharacterizationofamultiterminal
. rate-distortion function. Berger [6] and Tung [7] extended the Slepian-Wolf problem to the lossy distributed encoding of an
1
0 arbitrary number of correlated sources. In those two works, inner and outer bounds were presented for the multiterminal
7 rate-distortion region, which do not coincide in general. Berger and Yeung [8] addressed a variant of the Wyner-Ziv problem,
1 namely, the distributed encoding of two sources, one of which allows for lossy compression. Over the years, a significant
:
v progress has been made in the area of multiterminal source coding, including general frameworks for lossless compression
i [9], [10], results for Gaussian sources in various contexts [11], [12], and Wyner-Ziv networks containing an arbitrary number
X
of sources [13]. Yet, an exact solution to the multiterminal rate-distortion region remains open, even for the simplest scenario
r
considered in [6].
a
The idea that a decoder wishes to reproduce a primary source with the help of an auxiliary source, introduced by Wyner,
Ahlswede, and Ko¨rner, can be intuitively extended to an arbitrary number of auxiliary sources. Finding of the admissible
rate region of such a system defines the so-called many-help-one problem. This problem has been recognized as a highly
challenging one and only a few particular solutions are known to date. Ko¨rner and Marton [14] addressed a two-help-one
problem where the primary source is a modulo-two sum of correlated binary auxiliary sources. Gelfand and Pinsker [15]
determined the admissible rate region when the auxiliary sources are discrete, memoryless, and conditionally independent if
the primary source is given. Motivated by the Gelfand-Pinsker result, Oohama [16] determined the rate-distortion region for
the same setup but Gaussian sources. Tavildar [12] derived the rate-distortion region for Gaussian memoryless sources with a
correlation model following a tree-structure.
This work contributes new results to the many-help-one problem for lossless compression of the primary source. As in
TheworkofA.WolfwassupportedbytheFP7projectICT-619555RESCUE(Links-on-the-flyTechnologyforRobust,EfficientandSmartCommunication
inUnpredictable Environments) whichispartlyfundedbytheEuropeanUnion.
TheworkofM.Do¨rpinghauswassupportedinpartbytheDeutscheForschungsgemeinschaft(DFG)intheframeworkoftheclusterofexcellenceEXC1056
“Center forAdvancing Electronics Dresden(cfaed)” andbytheDFGwithin theCollaborative Research Center SFB912“HighlyAdaptive Energy-Efficient
Computing(HAEC)”.
A. Wolf, M. Do¨rpinghaus, and G. Fettweis are with the Vodafone Chair Mobile Communications Systems, Technische Universita¨t Dresden, Dresden,
Germany,E-mails:{albrecht.wolf,meik.doerpinghaus, fettweis}@tu-dresden.de.
D. C. Gonza´lez and J. C. S. Santos Filho are with the Department of Communications, School of Electrical and Computer Engineering, University of
Campinas (UNICAMP),Campinas–SP,Brazil, E-mails:{dianigon, candido}@decom.fee.unicamp.br.
2
[15] and [16], we consider that the auxiliary sources (a.k.a. helpers) are conditionally independent given the primary source.
This is equivalent to assume that the auxiliary sources represent independently degraded versions of the primary source, thus
finding direct application to emerging cooperative communication schemes with multiple lossy relaying routes from source
to destination. On the other hand, unlike [15] and [16], we focus on the binary case, to which our major contributions refer.
ThesecontributionsaredetailedinSectionI-F.We introducetwovariationsofthemany-help-oneproblem,denotedasfollows:
(i) the strong many-help-one problem, where not only the primary source must be recovered errorless, but also the auxiliary
sources themselvesmust be recoveredunder any given set of distortion constraints, and (ii) the weak many-help-oneproblem,
where only the primary source must be recovered, the auxiliary sources serving merely as side information.
X1 2nR1 Xˆ1 X1 2nR1 Xˆ1
Encoder1 Encoder1
(SPoruimrcaery1) Ehd1(X1,Xˆ1)i≤ǫ (SPoruimrcaery1) Ehd1(X1,Xˆ1)i≤ǫ
X2 2nR2 Xˆ2 X2 2nR2
Encoder2 Encoder2
(ASouuxrilciear2y) DJeocoindter Ehd2(X2,Xˆ2)i≤D2+ǫ (ASouuxrilciear2y) DJeocoindter
XN 2nRN XˆN XN 2nRN
EncoderN EncoderN
(SAouuxricliearNy) EhdN(XN,XˆN)i≤DN+ǫ (SAouuxricliearNy)
Receiver Receiver
(a) (b)
Fig.1:(a)Thestrongmany-help-one problem;(b)theweakmany-help-one problem.
A. The Strong Many-Help-One Problem
The strong many-help-oneproblem is presented in Fig. 1a. It has N discrete memorylesssources, denoted as {(X ,X ,
1,k 2,k
...,X )}∞ , with the n-sample sequence of the ith source being represented in vector form as X =[X ,X ,...,X ],
N,k k=1 i i,1 i,2 i,n
i = 1,...,N. When appropriate, for simplicity, we shall drop the temporal index of the sequences, denoting the sources
merely as X ,X ,...,X . By assumption, X ,X ,...,X are mutually correlated random variables, taking values in fi-
1 2 N 1 2 N
nite sets X ,X ,...,X , respectively, and are distributed according to a fixed and known probability mass function (pmf)
1 2 N
p(x ,x ,...,x ).TheencodersgeneratebinarysequenceswithratesR ,R ,...,R bitsperinputsymbol.Thedecoderoutput
1 2 N 1 2 N
is a sequence of N-tuples {(Xˆ ,Xˆ ,...,Xˆ )}∞ whose components take values from finite reproduction alphabets
1,k 2,k N,k k=1
Xˆ ,Xˆ ,...,Xˆ . The encodingis donein blocksof lengthn andthe distortionconstraintis givenforthe primarysourceX by
1 2 N 1
n
1
E d (X ,Xˆ ) ≤ǫ (1)
1 1,k 1,k
n
" #
k=1
X
and for the auxiliary sources X ,i=2,...,N, by
i
n
1
E d (X ,Xˆ ) ≤D +ǫ, (2)
i i,k i,k i
n
" #
k=1
X
where d (x ,xˆ ) ≥ 0,x ∈ X ,xˆ ∈ Xˆ, is a given distortion function which can differ for each source. Moreover, E[·]
i i i i i i i
denotesthe expectationoperatorandǫis asmallpositivenumber.Thestrongmany-help-oneproblemcanbeseenasa discrete
multiterminal source coding with N −1 distortion constraints assigned to the auxiliary sources.
B. The Weak Many-Help-One Problem
The weak many-help-oneproblemis presentedin Fig. 1b. It is similar to the strong many-help-oneproblem,exceptthat the
decoder output is only the sequence {Xˆ }∞ whose componentstake values from the finite reproductionalphabet Xˆ . The
1,k k=1 1
encoding is done in blocks of length n and the only distortion constraint is given for the primary source X by
1
n
1
E d (X ,Xˆ ) ≤ǫ. (3)
1 1,k 1,k
n
" #
k=1
X
The auxiliary sources X ,i=2,...,N, are not reconstructed at the receiver.
i
3
C. Independently Degraded Helpers
In this work, following [15] and [16], we consider that the auxiliary sources (a.k.a. helpers) are conditionally independent
given the primary source. Hereafter, this scenario is referred to as the CI condition. Thus the joint probability distribution of
X ,X ,...,X satisfies
1 2 N
N
p(x ,x ,...,x )=p(x ) p(x |x ). (4)
1 2 N 1 i 1
i=2
Y
As mentioned before, the CI condition finds important application to emerging cooperative communication techniques based
on lossy relaying links.
D. Problem Statement
We define the rate-distortionregionR(D ,...,D ) for the strong many-help-oneproblemand the admissible rate region R
2 N
for the weak many-help-one problem as the set of rate N-tupels R ,R ,...,R for which the systems in Fig. 1a or Fig. 1b
1 2 N
satisfy the averagedistortionsconstraintsin (1) and (2), or in (3), as required,for n→∞, respectively.The essential problem
is to characterize the set of minimum rate N-tuples at which the encoders can communicate with the decoder while still
conveying enough information to satisfy the distortion constraints on the reconstruction.
E. Known Results
a) Strong many-help-oneproblem: Berger [6] and Tung [7] studied the problem of N memoryless discrete sources with
arbitrary distortion constraints D ,D ,...,D . In [6, Theorem 6.1 and Theorem 6.2], an inner bound, i.e., an achievable
1 2 N
rate-distortion region, R (·)⊆R(·), and an outer bound R (·)⊇R(·) were provided, respectively. Eventually, the inner and
a c
the outer bound do not coincide and thus the rate-distortion region remains unknown for the referred problem.
Gastpar[13]extendedtheBerger-Tungproblembyprovidinguncodedsideinformationatthedecoder.Alternatively,Gastpar’s
work can be seen as an extension of the Wyner-Ziv problem [5] to N sources. In [13, Theorem 2 and Theorem 3], an inner
and an outer bound were providedfor the rate-distortion region, respectively. However, similarly to the Berger-Tung problem,
thoseboundsdonotcoincide.Thisisnotsurprising,indeed,astheanalysisin[13]isanextensionoftheBerger-Tungproblem.
In [13, Section V], a partial converse was presented, with the assumption that all the sources are conditionally independent
given the side information. It was shown in [13, Theorem 6] that in this case the inner and the outer bound coincide.
Berger and Yeung [8] investigated the Berger-Tung problem when specialized to two memoryless discrete sources with
distortion constraints D =0 and D is arbitrary.Note that this setup correspondsto the strong one-helps-oneproblem.In [8,
1 2
Section III and Section IV], inner and outer bound were provided, respectively. Once again, the two bounds do not coincide.
Additionally, in [8, Section VI], binary sources were considered which are interconnected via a binary symmetric chennel
(BSC) and the distortion was measured by the Hamming distance. It was then shown that, depending on D , the inner and
2
the outer bound can coincide.
Bounds on the rate-distortion region of the strong many-help-one problem in Fig. 1a can be derived by specializing the
resultsin [6], [7],and [13], i.e.,bysetting a zerodistortionconstraintforsourceone,thatis, D =0. However,nopublication
1
so far addressed the problem of conditionally independentauxiliary sources given the primary source. Moreover,boundswith
binary sourcesand Hamming distortionmeasure are consideredin neither [6], [7], nor [13]. Berger and Yeung [8] did address
binary sources and Hamming distortion measure, but only for two sources.
b) Weak many-help-one problem: Gelfand and Pinsker [15] determined the admissible rate region for a discrete lossless
version of the so-called CEO (Chief Executive Officer) problem, under the same CI condition assumed herein. Note that the
weak many-help-one problem is a special case of the lossless CEO problem. However, Gelfand and Pinsker did not address
binary sources and the Hamming distortion measure, for which we shall obtain insightful particular results.
F. Contributions of this Work
The contributions of this paper include the following:
1) InSectionII-AandSectionII-B,weprovideboundsontherate-distortionregionR(D ,...,D )forthestrongmany-help-
2 N
oneproblem.Morespecifically,aninnerboundR (D ,...,D )⊆R(D ,...,D )andanouterboundR (D ,...,D )⊇
a 2 N 2 N c 2 N
R(D ,...,D ) are given based on the results in [13]. As in [13], our bounds do not coincide, which is not surprising,
2 N
since we address is a special case of the Berger-Tung problem. Our bounds generalize those presented in [17] for two
sources.
2) We investigate the CI case of the strong many-help-one problem, where the auxiliary sources are independent while
conditioned on the primary source in Section II-D. The inner and the outer bound do not coincide for this case. This
4
result differs from that presented in [13, Section V], where the inner and the outer bound coincide if all the sources are
conditionally independentgiven uncoded side information.Note, however, that the conditionalindependence in [13] also
differs from the one we consider, as in the former the conditioning variable is not the primary source, but some side
information.
3) In Section II-E, we derive single-letter expressions for the inner and the outer bound of the rate-distortion region in 2)
with binary sources and Hamming distortion measure. We show, that depending on the distortion constraints D ,...,D ,
2 N
the inner and the outer bound can coincide.
4) In Section III-A, based on the results from Gelfand and Pinsker [15], we derive the admissible rate region of the weak
many-help-oneproblem.Furthermore,wederivesingle-letterexpressionsfortheadmissiblerateregionwithbinarysources,
CI condition, and Hamming distortion measure in Section III-B. We show that independent decoders for the auxiliary
sources are optimal, which is a remarkable result having operational meaning.
G. Notation and Terminology
The upper- and lowercase letters are used to denote random variables and their realizations, respectively. The alphabet set
of a random variable X with realization x is denoted by X, and its cardinality, by |X|. The pmf of the random variable X
is denoted by p (x), or simply p(x) when this does not create any confusion. Also, X and x represent vectors containing a
X
temporalsequenceofX andx,respectively.Allrandomvectorshavelengthn(blocklength).Weusektodenotethetimeindex
anditodenoteasourceindex.WedefineA ,{A |i∈S}asanindexedseriesofrandomvectors,A (b ),{A (b )|i∈S}
S i S i i i
as an indexed series of random time vectors with variable b and A ,{A |i∈S} as an indexed series of random variables.
i S i
In general, a vector A contains elements a , as in A = {a ,a ,...,a }. We define two particular sets: N = {1,2,...,N}
(·) 1 2 |A|
and L={2,3,...,N}.The binaryentropyfunction is denoted as h(p)=−plog(p)−(1−p)log(1−p). The probabilityof an
event E is denoted as Pr[E]. The binary convolution is defined as a ∗a =a (1−a )+(1−a )a . The multivariate binary
1 2 1 2 1 2
convolution is defined as a ∗...∗a =a ∗(...∗(a ∗a )...), which is a cascaded binary convolution.
1 N 1 N−1 N
Moreover, we shall make use of the concept of strong typicality as introduced in [6]. We use the definition as in [18, p.
326].LetN(a|x) bethe numberofoccurrencesofa symbola in thesequencex. Thesequencex issaidto beǫ-strongtypical
with respect to a pmf pX(x) if, ∀a ∈ X, |1/nN(a|x)−pX(a)| < ǫ/|X|. For a given pmf pX(x), the set of ǫ-strong typical
sequences will be denoted by T∗(n)(X), or simply T∗(n).
ǫ ǫ
II. THESTRONG MANY-HELP-ONEPROBLEM
Inthissectionweinvestigatethestrongmany-help-oneproblemdepictedinFig.1aandprovideaninnerandanouterbound
for the rate-distortion region.
A. An Inner Bound
An inner bound can be obtained based on the coding scheme introduced by Berger in [6]. In this scheme, the sequence X
i
is encoded in two stages. First, a code is used with a suitable quantization vector referred to as codeword; second, a binning
operation for the codeword indices is applied. Herein, differentfrom [6], encoder one has to meet a zero distortion constraint,
i.e., D = 0, and thus the quantization is tightly restricted, if possible at all. All encoders use a joint codebook, but operate
1
independently. Each encoder applies a binning operation with respect to the codeword index. Based on the bin index, the
decoder aims at the perfect reconstruction of codewords. This requires all reconstructed codewords to be jointly typical.
Theorem 1: R (D )⊆R(D ), where R (D ) is the set of all rate N-tuples R such that there exists a N −1-tuple U
a L L a L N L
of discrete random variables with
N
p(u ,x )=p(x ) p(u |x ) (5)
L N N i i
i=2
Y
for which the following conditions are satisfied:
H(X |U )+I(X ;U |X ,U ) if 1∈S, (6a)
1 Sc S\{1} S\{1} 1 Sc
R ≥
i
I(X ;U |X ,U ) otherwise, (6b)
i∈S (cid:26) S S 1 Sc\{1}
X
∀S ⊆N, with Sc being the complement of S; and for which there exist functions g (·),∀i∈N, such that for
i
E[d (X ,g (X ,U ))]≤ǫ (7)
1 1 1 1 L
and ∀i∈L
E[d (X ,g (X ,U ))]≤D +ǫ. (8)
i i i 1 L i
Proof: See Appendix A. (cid:4)
5
B. An Outer Bound
We now present a general outer bound R (D ) which contains the desired rate-distortion region R(D ), i.e., R (D ) ⊇
c L L c L
R(D ). Our derivation follows the standard outer-bound arguments given in [6, Theorem 6.2].
L
Theorem 2: R (D )⊇R(D ), where R (D ) is the set of all rate N-tuples R such that there exists a N −1-tuple U
c L L c L N L
of discrete random variables whose joint pmf p(u ,x ) must satisfy the marginal constraints
L N
p(u ,x )=p(x )p(u |x ), (9)
L N N i i
uj,jX∈L\{i}
∀i∈L, for which the following conditions are satisfied:
H(X |U )+I(X ;U |X ,U ) if 1∈S, (10a)
1 Sc L S\{1} 1 Sc
R ≥
i
I(X ;U |X ,U ) otherwise, (10b)
i∈S (cid:26) L S 1 Sc\{1}
X
∀S ⊆N; and for which there exist functions g (·),∀i∈N, such that
i
E[d (X ,g (X ,U ))]≤ǫ (11)
1 1 1 1 L
and ∀i∈L
E[d (X ,g (X ,U ))]≤D +ǫ. (12)
i i i 1 L i
Proof: See Appendix B. (cid:4)
C. Discussion
Since R and R do not coincide in general, the exact rate-distortion region is unknown for the strong many-help-one
a c
problem in Fig. 1a. The most important distinction between R in Theorem 1 and R in Theorem 2 is that one between
a c
the condition on pmf p(x ,u ) in (5), i.e., U → X → X → U , and the condition on pmf p(x ,u ) in (9), i.e.,
N L i i j j N L
U → X → X ,X → X → U , ∀i ∈ L, ∀j ∈ L and j 6= i, respectively. The former corresponds to the inner bound
i i j i j j
approach that the encoders are assumed to operate independently, i.e., the joint conditional inner-bound pmf must factor. In
contrast, the outer bound approachassumes that encoderscan operate dependentlyon each other, which is a weaker condition
on the outer-bound pmf.
ThepresentedinnerandouterboundinTheorem1andTheorem2coincidewithknownrateregionsundercertainconditions:
• If Di →0,i=1,...,N, Ra(DL) and Rc(DL) coincide and reduce to the Slepian-Wolf rate region in [1].
• If only two sources are available Ra(DL) and Rc(DL) coincide with the inner and the outer bound presented in [8].
• If R1 ≥H(X1), Ra(DL) and Rc(DL) coincide with the inner and the outer bound presented in [13].
D. Independently Degraded Helpers
In this section we present an inner and an outer bound for the rate-distortion region under the CI condition. The results
herein are along the line of the reasoning in [13, Section V]. Eventually, we end up with a similar formulation, except that in
ourcase the auxiliarysourcesare conditionallyindependentgiventhe primarysource,whichis encodedwith rate R , whereas
1
in [13, Section V] the sources are conditionally independent given an side information, which is available at the receiver and
not encoded. Due to this difference, our rate region R (D ) and R (D ) do not coincide in general and are only tight for
a L c L
specific constraints, as opposed to [13, Section V], where the inner and the outer bound are tight in general.
The CI assumption allows us to simplify the inner and the outer bound on the rate-distortion region, as follows.
Corollary 3: If X ,X ,...,X are conditionally independent given X , i.e., X ∼ (4), then
2 3 N 1 N
R (D )⊆R(D ),
a L L
where R (D ) is the set of all rate N-tuples R such that there exists a N −1-tuple U of discrete random variables with
a L N L
N
p(u ,x )=p(x ) p(u |x )·p(x |x ) (13)
L N 1 i i i 1
i=2
Y
6
for which the following conditions are satisfied:
R ≥H(X |U )+ I(X ;U |X ) if 1∈S, (14a)
i 1 Sc i i 1
 Xi∈S i∈XS\{1}
 Ri ≥I(Xi;Ui|X1) ∀i∈L, (14b)
∀S ⊆N; and for which there exist functions gi(·),∀i∈N, such that
E[d (X ,g (X ,U ))]≤ǫ (15)
1 1 1 1 L
and ∀i∈L
E[d (X ,g (X ,U ))]≤D +ǫ. (16)
i i i 1 i i
Proof: See Appendix C. (cid:4)
Note that g (·), ∀i∈L is only a function of X and U , in contrast to g (·) in (8) where the function is dependent on X
i 1 i i 1
and all U . The function g (·) is still a function of X and U .
L 1 1 L
Corollary4:IfX ,X ,...,X areconditionallyindependentgivenX ,i.e.,X ∼(4),thenR′(D )⊇R (D ),andhence
2 3 N 1 N c L c L
R′(D ) ⊇ R(D ), where R′(D ) is the set of all rate N-tuples R such that there exists a N −1-tuple U of discrete
c L L c L N L
random variables whose joint pmf p(u ,x ) must satisfy the marginal constraints
L N
p(u ,x )=p(x )p(u |x ), (17)
L N N i i
uj,jX∈L\{i}
∀i∈L, for which the following conditions are satisfied:
R ≥H(X |U )+ I(X ;U |X ) if 1∈S, (18a)
i 1 Sc i i 1
 Xi∈S i∈XS\{1}
 Ri ≥I(Xi;Ui|X1) ∀i∈L, (18b)
∀S ⊆N; and for which there exist functions gi(·),∀i∈N, such that
E[d (X ,g (X ,U ))]≤ǫ (19)
1 1 1 1 L
and ∀i∈L
E[d (X ,g (X ,U ))]≤D +ǫ. (20)
i i i 1 i i
Proof: See Appendix D. (cid:4)
Note that g (·), ∀i∈L is only a function of X and U in contrast to g (·) in (12) where the function is dependent on X
i 1 i i 1
and all U . The function g (·) is still a function of X and U .
L 1 1 L
TheinnerboundR (D )in(14a),(14b)andtheouterboundR (D )in(18a),(18b)looksimilar,exceptthattheunderlying
a L c L
auxiliary random variables U in Corollary 4 have more degrees of freedom, as pointed out in Section II-C. For (18b), each
L
mutual information I(X ;U |X ) depends only on the marginal distribution of (X ,U ,X ), ∀i ∈ L. Hence the additional
i i 1 i i 1
degreesoffreedomcannotlowerthemutualinformation.Therefore,theconstraintsontheratesin(14b)and(18b)areequivalent.
However, this statement is not true for the constraints in (14a) and (18a). In Corollary 4, the additional degrees of freedom
for the auxiliary random variables U can result in a smaller conditional entropy H(X |U ) in (18a) in comparison to the
L 1 Sc
conditionalentropy H(X |U ) in (14a) of Corollary 3. In conclusion, the rate regionsin Corollary 3 and Corollary 4 do not
1 Sc
coincide in general, but they do coincide for (14b) and (18b).
Remark: If R ≥H(X ), all sum rate bounds in (14a) and (18a) are implicitly satisfied by the lower single rate bounds of
1 1
(14b) and (18b), respectively, i.e., all single rates R ≥ I(X ;U |X ). Hence, in this case R (D ) = R (D ), as presented
i i i 1 a L c L
in [13, Theorem 6].
7
X1 2nR1 Xˆ1
Encoder1
(SPoruimrcaery1) EhdH(X1,Xˆ1)i≤ǫ
X2 2nR2 Xˆ2
BSC(p2) Encoder2
(ASouuxrilciear2y) DJeocoindter EhdH(X2,Xˆ2)i≤D2+ǫ
XN 2nRN XˆN
BSC(pN) EncoderN
(SAouuxricliearNy) EhdH(XN,XˆN)i≤DN+ǫ
Receiver
Fig.2:Thestrongmany-help-one problemwithbinarysources,CIcondition, andHammingdistortion measure.
E. Binary Sources and Hamming Distortion
We now consider a special case of the strong many-help-one problem, with binary sources, CI condition, and Hamming
distortion measure illustrated in Fig. 2. Suppose that the random variable X takes values from a binary set X ={0,1} with
1
uniformprobabilities,i.e.,Pr[X =0]=Pr[X =1]=0.5.Assume thatX is theinputand X isthe outputof a memoryless
1 1 1 i
BSC with cross-over probability p ≤ 0.5, ∀i ∈ L. All BSCs are independent, i.e., p(x ,...,x |x ) = N p(x |x ), and
i 2 N 1 i=2 i 1
the errors occurring in X , ∀i ∈ L are i.i.d., i.e., p(x |x ) = n p(x |x ). Thus, the auxiliary sources X ,∀i ∈ L, are
i i 1 k=1 i,k 1,k Q i
conditionally independent given the primary source. The distortion constraints are given by (1) and (2), where the distortion
Q
is measured by the Hamming distance, i.e.,
1, if x 6=xˆ ,
i,k i,k
d (x ,xˆ )= (21)
H i,k i,k
(0, if xi,k =xˆi,k.
In this section we provide an inner and an outer bound for the associated rate-distortion region. Single-letter expressions
are found for the mutual information I(X ;U |X ), ∀i ∈ L and for the conditional entropy H(X |U ) in Corollary 3 and
i i 1 1 Sc
Corollary 4, in terms of the cross-over probability p and the distortion constraint D , ∀i ∈ L. To derive minI(X ;U |X )
i i i i 1
where the minimumis takenoverall p(u |x ) suchthat E[d (X ,g (X ,U ))]≤D +ǫ, we applyknownresultsfrom [5] and
i i i i i 1 i i
[18, Theorem 10.3.1]. Depending on the rate R , we need to distinguish between two cases for the sake of completeness, as
i
shall become apparent soon. In case one, referred to as joint decoding, each auxiliary source X is decoded with the help of
i
the primary source X , i.e., I(X ;U |X ) is at play; in case two, referred to as independent decoding, the auxiliary sources
1 i i 1
are decoded without the help of the primary source, i.e., I(X ;U ) is at play. In Lemma 5 we derive a single-letter expression
i i
for minH(X |U ), where the minimum is taken over all p(u |x ) such that E[d (X ,g (X ,U ))]≤ǫ, based on [19].
1 Sc Sc 1 H 1 1 1 Sc
1) Mutual Information I(X ;U |X ):
i i 1
a) Joint Decoding: If the auxiliary source X is decoded with the help of the primary source X , we can consider this
i 1
as the Wyner-Ziv problem with side information at the decoder [5]. Here the primary source acts as the side information and
is available at the receiver side. In [5, Section II] the binary case with uniform probabilities and Hamming distortion measure
was investigated. From [5, Section II], the mutual information I(X ;U |X ) is given as
i i 1
f(D ) for 0≤D ≤D ,
R′(D )=I(X ;U |X )= i i c (22)
i i i i 1 ((pi−Di)f′(Dc) for Dc <Di ≤pi,
with f(D )=h(p ∗D )−h(D ), f′(x)= ∂f(x), and D isthe solutionto the equationf(D )/(p −D )=f′(D ). In Fig.3
i i i i ∂x c c i c c
the rate-distortionfunctionof the Wyner-Zivproblemwith side informationat the decoderis presented.Additionally,an upper
bound h(p ∗D )−h(D ) and a lower bound h(p )−h(D ) on R′(D ) are shown. From now on, we refer to R′(D ) as the
i i i i i i i i i
joint decoding rate function. Please note that D is upper bounded by p .
i i
So far we considered the reconstruction of the auxiliary source X under distortion constraint D . However, if encoder i
i i
provides a rate larger than 1−h(D ) a distortion constraint d smaller than D can be achieved by decoding the auxiliary
i i i
source X without any side information. In what follows we discuss this case.
i
b) Independent Decoding: We consider the case in which the auxiliary source X is decoded without the help of the
i
primarysource. We define U (d ), 0≤d ≤0.5, to be a binaryrandomvariable obtainedby connectionto X via a BSC with
i i i i
8
cross-overprobabilityd . The auxiliarysource X is decoded withoutany side informationat the receiver,i.e., d is not upper
i i i
bounded by p as in (22). From [18, Theorem 10.3.1], the mutual information I(X ;U ) is given as
i i i
R (d )=I(X ;U )=1−h(d ) for 0≤d ≤0.5. (23)
i i i i i i
From now on, we refer to this function as independent decoding rate function, shown in Fig. 3. The mutual information
R′(D ) = I(X ;U |X ) in (22) is always upper bounded by the mutual information R (d ) = I(X ;U ) in (23) for equal
i i i i 1 i i i i
distortion constraints.
In summary, we found single-letter expressions for the mutual information in (14a), (14b), (18a), and (18b), in terms of
the cross-over probability and the distortion constraint. Depending on the rate of encoder i, a distinction between joint and
independent decoding is necessary. We now derive a single-letter expression for the conditional entropy H(X |U ) in (14a)
i Sc
and (18a).
1
Ri(di)=1−h(di)
Ri′(Di)
h(pi)−h(Di)
h(pi∗Di)−h(Di)
h(pi)
f(Dc)
h(pi∗pi)
−h(pi)
Di,di
Dc pi 0.5
Fig.3:Jointdecoding ratefunctionbasedontheWyner-Zivtheorem andindependent decoding ratefunctionbasedontherate-distortion theory.
2) Conditional Entropy H(X |U ): From the auxiliary sources X ,∀i ∈ Sc, U is made available at the receiver, and
1 Sc i Sc
we aim to reconstruct X perfectly. From (16) and (20) we know that the function g (·), ∀i ∈ Sc is only dependent on X
1 i 1
and U . In other words, with the CI condition optimal performance is achieved if all auxiliary source encoders and decoders
i
operate independently of each other. As H(X |U ) expresses the remaining uncertainty of X when U is known, we can
1 Sc 1 Sc
evaluate H(X |U ) by assuming that U ,∀i ∈ Sc is decoded independently of X at the receiver. The problem of perfect
1 Sc i 1
reconstructionof source X with given side informationU correspondsto the sourcecoding problemwith side information,
1 Sc
for which the rate region is given in [19]. In [19, Section II] it is shown that minH(X1|USc) is achieved if the primary
source encoder operates independently from the encoder processing the side information. In our case the side information is
U processed by encoders i for i∈Sc. Thus, if the auxiliary random variables U satisfy
Sc Sc
p(x ,x ,u )=p(x ,x ) p(u |x ), (24)
1 Sc Sc 1 Sc i i
i∈Sc
Y
we achieve minH(X |U ), where the minimum is taken over all p(u |x ) such that E[d (X ,g (X ,U ))] ≤ ǫ. In
1 Sc Sc 1 H 1 1 1 Sc
summary, we can make the following statements for the conditional entropy H(X |U ) considering binary sequences, CI
1 Sc
condition and the Hamming distortion measure.
1) minH(X |U ) is achieved if all encoders operate independently.
1 Sc
2) As a consequenceof 1), the inner boundin Corollary3 andthe outer boundin Corollary4 forminH(X |U ) coincide.
1 Sc
3) Auxiliary source decoders operate independently.
We now derive the single-letter expression for the conditional entropy.
Lemma5:LetX ,X ,...,X beconditionallyindependentgivenX whichtakesvaluesfromthebinarysetX ={0,1}
v1 v2 v|V| 1
with uniform probabilities. The dependency between X and X is given by the cross-over probability p ∈ [0,0.5] of a
1 v v
BSC, for v ∈V and arbitrary set V. The binary auxiliary random variables U are constructed according to the pmf in (24).
V
Furthermore,sources X , v ∈V, are decoded without any side information at the receiver,i.e., the rate-distortionfunction for
v
9
eachsourceX ,v ∈V,isgivenby(23).ThedependencybetweenX andU isgivenbythedistortionconstraintd ∈[0,0.5],
v v v v
for v ∈V. Moreover, the rate function for the primary source is
R = min I(X ;Xˆ )= min H(X |U )=φ(p ,d ), (25)
1 1 1 1 V V V
p(xˆ1|x1):E[dH(x1,xˆ1)]≤ǫ p(uV|x1):E[dH(x1,g1(x1,uV))]≤ǫ
where φ(p ,d ) is the single-letter expressionof H(X |U ) in terms of cross-overprobabilitiesp and distortionconstraints
V V 1 V V
d given in (28).
V
Proof: minH(X |U ) is achievedfor independentencodersas arguedabove,i.e., X ,U ∼(24). Thus, we can rewrite the
1 V 1 V
conditional entropy in (25) as follows
minH(X |U )=H(X ,U )−H(U ) (26)
1 V 1 V V
=H(X )+ H(U |X )− H(U )+ I(U ;U )
1 v1 1 v1 v1 v2
{vX1}∈V {vX1}∈V {v1X,v2}∈V
− I(U ;U ;U )+...± I U ;...;U . (27)
v1 v2 v3 v2 v|V|
{v1,vX2,v3}∈V {v1,..X.,v|V|}∈V (cid:0) (cid:1)
If V =6 {∅}, then
minH(X |U )= h(p ∗d )− h(p ∗d ∗p ∗d )
1 V v1 v1 v1 v1 v2 v2
vX1∈V {v1X,v2}∈V
+ h(p ∗d ∗p ∗d ∗p ∗d )
v1 v1 v2 v2 v3 v3
{v1,vX2,v3}∈V
−...± h p ∗d ∗...∗p ∗d (28)
v1 v1 v|V| v|V|
{v1,..X.,v|V|}∈V (cid:0) (cid:1)
,φ(p ,d ). (29)
V V
If V ={∅}, then
minH(X |∅)=H(X )=1,φ(∅,∅). (30)
1 1
The steps are justified as follows:(26) is the chainrule of entropy;for(27) we haveused the chain rule of mutualinformation
and multivariatemutualinformation[20] and the factthat U ,U ,...,U are conditionallyindependentgivenX ; and (28)
v1 v2 v|V| 1
followsfromtheMarkovchainU →X →X ,withuseofH(U |X )=h(p ∗d )asprovenin[19].FromtheMarkovchain
v v 1 v 1 v v
U →X →X →X →U (followingfrom(24)) we havethe conditionalentropyH(U |U )=h(p ∗d ∗p ∗d )
vi vi 1 vj vj vi vj vi vi vj vj
for v ∈ V, v ∈ V and v 6= v . From the assumption on the pmf in (24) we also have the multivariate mutual information
i j i j
given as I(U ;...;U ) = H(X )−h(p ∗d ∗...∗p ∗d ). After some algebraic manipulations omitted here we
v1 v|V| 1 v1 v1 v|V| v|V|
finally arrive at (28). This completes the proof. (cid:4)
In what follows we have to carefully distinguish between jointly decoding the auxiliary sources X with “side information”
i
X and independently decoding the auxiliary sources X without “side information” X , ∀i ∈ L. Eventually, we have to
1 i 1
determine which decoding strategy minimizes and maximizes the inner and the outer bound — including both the conditional
entropy and the mutual information — in (14a), (14b), (18a), and (18b).
3) Outer Bound: We now derive a single-letter expression for Corollary 4 with binary source, CI condition, and Hamming
distortion measure.
Theorem 6: If X takes values from a binary set with uniform probabilities, X ∼p(x )=p(x ) N p(x |x ) and the
1 N N 1 i=2 i 1
binary random variables U are distributed with pmf (17), then it holds that
L
Q
R (D )⊇R(D ),
c L L
where R (D ) is the set of all rate N-tuples R for which the following conditions are satisfied:
c L N
R ≥φ(p ,d )+ R′(D ) if 1∈S, (31a)
i Sc Sc i i
 Xi∈S i∈XS\{1}
 Ri ≥Ri∗(di,ρi) ∀i∈L, (31b)
where

1−h(d ) for 0≤d <D , (32a)
i i i
Ri∗(di,ρi)=  argmax Rc(DL) for 0≤ρi ≤1,Di ≤di ≤0.5, (32b)
 Ri=ρi·(1−Rh(iD=i1)−)+h((d1−i)ρi)·R′i(Di)

10
∀i∈L.
Proof: Assume that a D -admissible code exists. We can make the following statements.
i
• The mutual information I(Xi;Ui|X1) in (18a), Corollary 4, is at least Ri′(Di), as shown in Section II-E1, Paragraph a)
i.e., the auxiliary source is decoded with the primary source.
• The conditional entropy H(X1|USc) in (18a), Corollary 4, is at least φ(pSc,dSc) as proved in Lemma 5, (25).
• For the mutual information I(Xi;Ui|X1) in (18b), Corollary 4, two decoding strategies can be considered. Other than
the mutual information in (18a)1, the mutual information in (18b) can either be determined by joint decoding, i.e.,
I(X ;U |X ) = R′(D ) (Section II-E1, Paragraph a)), or independent decoding, i.e., I(X ;U ) = 1−h(d ) (Section
i i 1 i i i i i
II-E1, Paragraph b)). Ultimately it is crucial which decoding strategy maximizes R (D ). Thus the mutual information
c L
in (18b) is at least R∗(d ,ρ ).
i i i
This completes the proof. (cid:4)
Note that in (31a) the underlyingdistribution of U is (24) as discussed in Lemma 5. The function R∗(d ,ρ ) is split into
Sc i i i
two regions:
• (32a): If the rate is Ri∗ > 1−h(Di), a distortion constraint di that is inferior to Di can be achieved by independent
decoding, i.e., I(X ;U ), such that R∗ = 1−h(d ) for 0 ≤ d < D . In other words, independent decoding maximizes
i i i i i i
the outer bound R (D ).
c L
• (32b):Iftherateis1−h(Di)≥Ri∗ ≥Ri′(Di)jointdecoding,i.e.,I(Xi;Ui|X1),orindependentdecoding,i.e.,I(Xi;Ui),
maximizes the outer bound R (D ). If joint decodingmaximizes R (D ), then R∗(·,ρ )=ρ ·(1−h(D ))+(1−ρ )·
c L c L i i i i i
R′(D ) and if independent decoding maximizes R (D ), then R∗(d ,·)=1−h(d ).
i i c L i i i
Note that ρ and d are continuous parameters that directly affect the outer bound R (D ). In Section II-E5 we discuss and
i i c L
illustrate results of the outer bound for two and three sources.
4) Inner Bound: We now derive a single-letter expression for Corollary 3 with binary source, CI condition, and Hamming
distortion measure.
Theorem 7: If X takes values from a binary set with uniform probabilities, X ∼p(x )=p(x ) N p(x |x ) and the
1 N N 1 i=2 i 1
binary random variables U are distributed with pmf (13), than it holds that
L
Q
R (D )⊆R(D ),
a L L
where R (D ) is the set of all rate N-tuples R for which the following conditions are satisfied:
a L N
φ(p ,d ) for 0≤d <D ,∀i∈L, (33a)
L L i i
R1 ≥ ( arg conv{ξ(pL,dQc,DQ,ρQ)} for 0≤di <Di,∀i∈Qc,ρi ={0,1},∀i∈Q, (33b)
R1
1−h(d ) for 0≤d <D ,∀i∈Qc, (34a)
i i i
R ≥
i ρ ·(1−h(D ))+(1−ρ )·R′(D ) for 0≤ρ ≤1,∀i∈Q, (34b)
(cid:26) i i i i i i
∀Qc ⊂L, withQ beingthecomplementofQc. {ξ(·)}={R ,R ,R }representsallknownachievableN-ratetuples,given
1 Qc Q
in (37). Furthermore, conv is a Simplex, introduced in [21, Section 2.2.4], defined as
C =conv{v ,...,v }= θ v +...+θ v θ ≥0, θ =1 . (35)
1 k 1 1 k k i i
( )
(cid:12) Xi
(cid:12)
Proof: We know achievable rate N-tuples RN in Corollary 3 for partic(cid:12)ular rate N −1-tuples RL. The mutual information
in (14b) is either
• Ri =I(Xi;Ui)=1−h(di), for 0≤di ≤Di,∀i∈Qc, i.e., (14b) = (34b) with ρi =1 (independent decoding); or
• Ri =I(Xi;Ui|X1)=Ri′(Di),∀i∈Q, i.e., (14b) = (34a) (joint decoding).
The achievable rate R can be determinant with the corresponding achievable rate sum bound in (14a). The underlying
1
assumption for the sum rate bound is that the primary source is jointly decoded with side information UQc and the auxiliary
sources i,∀i∈Q are jointly decoded with the primary source X . The achievable rate sum bound in (14a) is
1
R =H(X |U )+ I(X ;U |X )=φ(p ,d )+ R′(D ), (36)
i 1 Sc i i 1 Qc Qc i i
i∈{X1}∩Q iX∈Q iX∈Q
1Themutualinformation in(18a)isconsideredinasumratebound,thisassumesinprinciple ajointdecodingstrategy.