BSTJ 60: 10. December 1981: Source Coding for Multiple Descriptions II: A Binary Source. (Witsenhausen, H.S.; Wyner, A.D.)

4 MB·English

THE BELL SYSTEM
TECHNICAL JOURNAL

Source Coding for Multiple Descriptions
UA Binary Source

By H. 8. WITSENHAUSEN and A. D. WYNER

A uniformly distributed (id) binary source is encoded into two
binary da streams at rates, and R,, respectively. These eienees
are such thot by obwercing either one separately, a decoder cen
recover a xou cpprasimation of the source fat average ervor rates
D,, Dy respectively, and by obsereina Bath sequences, « decoler can
Blain e beter apprasimation of the saurce (at average era? Fate
Dy, in this paper a “converse” theorem ix extished vn these! of
‘achievable quincuples (Ps, Re. De, Ds, Dy). Fur the special eave Re =
y= 12, Dy 0, and D, D, our result implies that D= 1/5

1 wwtRopucTION

Let (eli be a sequonee of independent drawings af the binary
random variable-&, where Pr{X'=0) = Pe(%= 1) 1/2 Asse that
this sequence appears a ee uf Lmyrnbul per seco the output

fa data source. (Refer to Fig. L) An encoder oboerves this sequence
spd emits two binary sequenecs at rates fi, Me = 1. These seauences
fre auch that by observing either one, n devoder can reoover & oOd
Approximation fo the eource output wl by oerving both sequen,
8 deceder can obsain a botter approximavion Lo the source output
Letting Dy, Ds, and Dy bo the error rates which sult when the streams
trace Kj, rate Hy, and both stroanis are wee hy a dono, espe:
tively, our problem isto determine fin Une tial Sharon sense the

chievablo quintuples (Re, Ds, Dy, Ds). Our main vest ia
£° Uhearem which gives a neceieary condition an the achiev

ze
=p pel

ig. Communion ee,

ble quintuple, This paper extends a previous one on the same sub
fect! This paper, however, in sel-contained.

‘This problem isan dealzetion ofthe ituatir
to

(i send information ovar two eeparate chennele, as in a packet
communication networ, and

i) recover aq uch ofthe orgial information as posible, should
‘one of the channel break down,

"To ix ideas et us say that R, = Re = 1/2, Dy ~ 0, and Dy = Ds = D.
‘Th the souree sequence nt ale 1 ito be encoded nto two sequences
‘race 1/2 each, auch thatthe original sequence canbe recovered from
thege two enol sequences with approximately zero error rate (Le,
De= 0), Our question hen becomes: How well ean we reconsiuct the
source sequence from one ofthe encoded steam —tha i, wha ie ehe
Iinimim D? A simple-minded approach woud bro Tet the encoded
‘reams consist of alternate anuree symbols, which will allot Dy ~ 0
In chia case, D'~ 1/4, since By observing evory other source symbal
decoder vil ake an eror haf the time un (he emsaing symbol. Ts it
Dorsile todo beter” Gaal and Cover ave looked a this problem
land have & theorem which can be vee to show that we can make
‘Dm (e2—1)/2© 0207, Ina previous paper it was shown that (with
y= B,— 1/2, Dy=0) Dz 1/8. The naw raul given here specializes
to D= 1/5 = 0200. The exace determination of tho best D remains
an open problet.*

whieh tin desined

1 FORMAL STATEMENT OF PROBLEM AND RESULTS

Let B= (0,1), and let eats y),% y «B®, be che Hamming distance
between the binary Nevectore x, yi fe, dit, y) i the number of
posilions in which 2 and y do not agree. A code with parameters

in Ra, Wises prove lowly pied sole which ensure he
sous Ua ne the me

2282 THE BELL SYSIEM TECHNICAL JOURNAL, DECEMBER 186:
Mi, Mh, Ds, D, Ds) ia quintuple of mappings (fu fi Bu Bu 63)
whet,

fo PN OM) ae LE (a)
Be (152 0 My) oe BN 1,2 an)
(12 MDX eB) > BS, ae)

“The source output na random vector X uniformly ditsibated on
Define

Y= eh 0)
400, en)

amd
X= #{f00, 400) (ee)

‘Thon the average crear rates are

EaX, Yh, ae)

pia dram, ow
1
eevee.) we

We say hat a quintuple (Ry, Rs, ey dhe) ie achienable if, for
arbicany ¢ > 0, there exists, for NY aufficendy large, a code with
parameters (N, Mf, Me, Ds, Dy, D.), where M, = 2%!" g = 1, 2 and
Did. +6, ~0, 1,2. The relationship of this formaliem: co the
tyavem of Fig 1 should be leer. Our problem i the detormination of
the set of achievable quins, aed cur main roll 2 cover
‘theorem

‘Before stating our result, let us take a moment co state a positive
theorem by Bl Gamal and Cover as it apeciatine tour problem
‘Theorem 1: The quintuple (i, Rs, dy dy di} 28 ochienable if there
cxists @ quadruple of random variahter X, 2, Y,2 which take vuluen
In, such thar PriX'= 0} ~ Pr{X~ 1} ~ 1/2, and!

BidutX, 8) 5 a, («
RaylX, YI 24h ca)
Bal, 2) = ds, ep

and
REMY), (a)

Sounce cons 2988
R212), (sb)
Ry RB TER Y.2) 4102), (a)
where I(--) ie he usual Shannon information.
or the spuvial cate of Rs ~ Rs ~ 1/2, dh = 0, ican be shown that
dh = dp = (42 1)/2 = 0207 is acbioveble
‘We now state our converse result
‘Theorem 2 1] Ry, Re dd, i) is achieve, then in all eases

Wh y> 1 = Aida to)
other a 83,
at
nomena) -a(ates 2h),
and + 2 thon
weewee—nit)a(tsam- ME),

where

0, <a
noy-[Bruerce-anneaay 82800
: Xda
Aout in his apr a ale the bate. x ios
211 toe ee) by mee, wont prove
ela een are ade io wi =
vat 1) nha

afr

eat
a
hich implies thot d = 1/
I, PROOF OF THEOREM 2
We sar fom the standard identity
Tt Ue Uy = HU Ud + HUG THI), ”

for arbitcary sundown vatiables Ui, Us, Uy We say that, Ua, Us ina
“\turkor cin” i Ty, Us are conditionally independent given Ui: Le,
Undepensixon 1, Uzonly chvough Li If Uy, Us, Uris a Markov chain
then FU; Ea[ = 0, and from (2)

‘2204 THE BELL SYSTEM TECHNICAL JOURNAL, DECEMBER 1063
Us, U) = 1G Bs UH) ~ Ls Wo. @
[Note that the hypothesis for (8) holds when Uy isa function of Us. A
sequence (Uy) ina Markov ebain if, fr all,

Goethe Ud Ba Ba Basa

isa Markov chain,
Let us now suppuve that we are given a ode (ff a Be) Heh
parameters (N, Mh, Me, Dy, Dy, D,). We can site
log Af, + log Ms = HA fER)) + BA FERD) 0
= 10, AED + AER) AER)
CF, HARD + 1K KVR) — 10)
PHA A+R YM,
where (8) follows from tho fact Una f(X) tke its values in @ set of
cardinality M, (10) holds because the pair A(X! (Ris determined by
and (1) holds because &, Y, and Z depend on X only throu F(X)
{AX 30 that (8) appli
Now (1) getting close to (Se) inthe direct theorem.
In fact, using (8) twice, we can underbound I[F(X); fi] by
TL; 7). Now the components af the source vector X aro independent,
fan the components of either ¥ or Z were also independent, we
ould make use of slandard techniques to establish the necesaity of
(Ge), But alas, we cannot aemuine thet ether the (Y} nor chy) are
Independent, so thae another vuole i required. The hey ide ia the
definition of another random vortar'V ~ [Vy c++ Vel the componente
‘of which wre in fact leper.
For 1S 2= My define the at
Ags (50) =) = FO aa
Le the cardinality of Ac be ju. Let che random veetor V be defined by
ita conditional distribution given X:

Ives Ye Aja
°. ther
‘Thus, given Xe Ac, V is uniformly distibuted on Ay I (allows that

the unconditional distribution for Vis

PxqVavi=2" ves
nd the components of ¥ are indeperent, a8 deained* Furthermore,
7, F030), X, F00, Vis a Maroy ci, so Oral, using 18,

Pr(V =vIk =) 03)

mo i viet fia ae ash ute aha of bene,

SOURCE CODING 2286
TA fi00; 000) = HV, £00; £00, = Vs. 8)
Combining (11) snd (14), we btn

1 L 1 1
Jr a6 +E tog ate 2 Bvt) + 21068.

HU; 2h + 510%: 8)

W), [aauvi a), [BdatX,
aes [Ese

‘tap (1) follows from the “rate distorion bound" which states that
EU be random voctor w
1X), and C iam arbitrary binary random vector, chen 21

sly iiribted in (an are Vand
One

taal Fat, 0] ee Ret 49

‘We will now obtain an upper bound on & in teres of Dy and 2s, As
“warm up” lec ua observe that from she triangle inequality,

af bet 2) 5 MV. + EBA NY + BaD)
Now
FiAEM= DD, BDR=DK — 8
ath

EdlV, ¥) = 5 Bidets, YOIV = vIPHAV =v).

Now suppote that we are given V — we As. Then, Y= af. Since
Prive =o",

aav. ¥) = ¥ eMac]

ofy mix-siadeun=. a9
Tas
sewn, a»

2286 THE BELL SYSTEM TECHNICAL JOURNAL, CECEMBER 1081
SSuhseination of (18) into (Sa) yields that for achievable
(Ras de dd

By + Ry 22 Hdl +c) hla), 9
‘which ithe result reported in Ref.

‘We will now gslaiah 2 tighter bound on A, namely, for De +
2s),

oo chat (5) vie thas for achievable (Ry, dy,
2a

Re keea adh on— mas, a)

which (6b, the inequality required for Theorem 2.

Upper bound on &: We estabish inequity (20) 8 follows. Lat,
1S BEM, hefited. Let Ay be as defined in (12), and let its cardinal
7m Let the members of Asbo the N-vcetors {x}. Let y = ax.
Tou, when X= x € Ay then Y ~ y, Now, form au x N
array, A, with math row

ae = an Boy += Bath =n ® ¥, ey

hare @ denotes todule 2 vector wien, Thus amy = 1, whom the
nth coordina:es of x and y ae diferent and dqy~ 0, otherwise. Note
thet

Asean)

3)

cy

{a the fraction of Tin column maf A
Next, for 1 m 5 lel ro = poeta) be the value of 2 which
reals when X=, lel Be the 37 array with math eo

be

Ue, Bay +25 Bus = By OY a

SOURCE CODING 2287
‘Then,
Mat, 2 f00)

Had dete te)

i

1
mE Malta Bal

P(E = bn) Daal =
pec eel

se} § an + Dan ~ Basal
eh

1
2S Laan ~ bon + 2B nal ~ 2
PL (0 ~ ann}

Bad

23 Loan ~ dna + Bean = Dred

eittecadehem ho
tex

-

LS am 1 wy

nd 4 en hp (0) We conde eat 6 le aw ut
at ‘

Enda mine =n= Sink an
tinal, comier

E Elda(¥, 21/8 k]

qe

1g
Re,

1
2 bat, 21% = x0]
pl P

1S prauv.na lk =

Bp 3, Bde 10 IK = x0)

15 v.
= gS, Eeete ne) PAL = VIR =). C8)

2288 THE BELL SYSTEM TECHNICAL JOURNAL, DECEMBER 1981
Now, from (19) which defines ¥

a vik nape fi ved
pay vitae = fA,
han Cy

Leet, 17.00) =

1g

Botan ant

= yg J alte bad

ba) (= ayaa

Fuld - 50h 9)

‘Now muko the dependence of sy anit om k explicit by waiting so,
and ta, respectively. Then, on svemmeing over & (2) becomes

1

be te

BioX, YN =F rl fiXy =A} S

a Bonne om
hr Pi =e 00 = Sindy, 0) ee

S Pon, Biber — snl. (801

ok ets t=

inaly, 129) becomes

L ay
aE elie.

BE Pin Aifewtt — ted ttl — sulle (80h

We noe apply the fllowing inequity, che proot uf which is gen
Sectinn TY

Let S, 7 be random variables auch that 0S T'= J, and £(S}
D, #(\P~ 8) = Ds, with 2D, + Des | Then

2
=D:

{80 T)+ PO 81) = De BD on

SOURCE COONG 2208
Let 8, 7 be che random variables which take the value sn ts
reopoctively, with probability Pin, ), then (20) and (31) imply thet,
for 2+ Bish,
apt
Be

whieh, when aubetiated in (15) ives (6b, proving Theorem 2,

As D4 2, (92)

1, PROOF OF THE INEQUALITY
Define @(Ds, 2.) a4 the supremum of ESL - 7) + TU - $)}
overall distributions of (8, 7) om the unit aqosre (0, 1 for which
{S}-s Dy B{|T~ BI) = De
Theorems 3: (a) Por 3D, + Dy 1 one has
aot

ato, b9 = 2 +. PE

ith Q(0,L) = 1 () For 2D, + D, = Lone haw

ams,

"To establish this, introduce for Sx, yin (0, 1h, 9 x 1, the
fonction”

poa}is-n-y 69

+(“2-2) 6-94 [ Ey
GS-9) e-9- [525
(8, Thin [0 1P ia Be + y— 2x4 /(A — 91.
Pro Ter fed 5a main of ve T mam by see
Rigliytestcrin oat
irre ee

Fase (Hn) ona [gon sen

a7

and thie mania over al either = 0 oF 8
(a) Fors =o)

pete 4t gy 2D

emt

TF

2290 THE BELL SYSTEM TECHNICAL JOURNAL, DECEMBER 1081