Linear Error Correcting Codes with Anytime Reliability

19 Pages·2011·0.74 MB·English

Linear Error Correcting Codes with Anytime
Reliability
Ravi Teja Sukhavasi and Babak Hassibi
1 Abstract
1
0
2 We consider rate R = k causal linear codes that map a sequence of k-dimensional binary vectors
n

b b toasequenceofn-dimensionalbinaryvectors c ,suchthateachc isafunctionof b t .
e { t}∞t=0 { t}∞t=0 t { τ}τ=0
F Such a code is called anytime reliable, for a particular binary-input memoryless channel, if at each time
(cid:16) (cid:17)
7 instant t, and for all delays d d , the probability of error P ˆb =b decays exponentially in
o t dt t d
1 (cid:16) (cid:17) ≥ − | (cid:54) −
d, i.e., P ˆb =b η2 βnd, for some β > 0. Anytime reliable codes are useful in interactive
] t−d|t (cid:54) t−d ≤ −
T communicationproblemsand,inparticular,canbeusedtostabilizeunstableplantsacrossnoisychannels.
I
. Schulman proved the existence of such codes which, due to their structure, he called tree codes in [1];
s
c
however, to date, no explicit constructions and tractable decoding algorithms have been devised. In this
[

1 paper, we show the existence of anytime reliable “linear” codes with “high probability”, i.e., suitably
v
chosen random linear causal codes are anytime reliable with high probability. The key is to consider
6
2
time-invariant codes (i.e., ones with Toeplitz generator and parity check matrices) which obviates the
5
3 need to union bound over all times. For the binary erasure channel we give a simple ML decoding
.
2
algorithm whose average complexity is constant per time iteration and for which the probability that
0
1 complexity at a given time t exceeds KC3 decays exponentially in C. We show the efficacy of the
1
: method by simulating the stabilization of an unstable plant across a BEC, and remark on the tradeoffs
v
Xi between the utilization of the communication resources and the control performance.
r
a
I. INTRODUCTION
Shannon’s information theory, in large part, is concerned with one-way communication of a message,
that is available in its entirety, over a noisy communication network. There are increasingly many
Ravi Teja Sukhavasi is a graduate student with the department of Electrical Engineering, California Institute of Technology,
Pasadena, USA [email protected]
Babak Hassibi is a faculty with the department of Electrical Engineering, California Institute of Technology, Pasadena, USA
[email protected]
This work was supported in part by the National Science Foundation under grants CCF-0729203, CNS-0932428 and CCF-
1018927,bytheOfficeofNavalResearchundertheMURIgrantN00014-08-1-0747,andbyCaltech’sLeeCenterforAdvanced
Networking.
applicationssuchasnetworkedcontrolsystems[2]anddistributedcomputing[1]wherecommunicationis
not one-way but interactive. A networked control system is characterized by the measurement unit begin
separatedfromthecontrolunitbyacommunicationchannel.Theinteractivenatureofcommunicationcan
be understood by observing that the measurements to be encoded are determined by the control inputs,
which in turn, are determined by the encoded measurements received by the controller. Block encoding
of the measurements is not applicable anymore because the controller needs real time information about
the system so that an appropriate control input can be applied. This is especially critical when the system
being controlled is open loop unstable. Any encoding-decoding delay translates into the system growing
increasingly unstable. Hence, the desired reliability of communication is determined by the quality of
the control input needed to stabilize the system.
In the context of rate-limited deterministic channels, significant progress has been made (see eg., [3],
[4]) in understanding the bandwidth requirements for stabilizing open loop unstable systems. When the
communicationchannelisstochastic,[5]providesanecessaryandsufficientconditiononthecommunica-
tion reliability needed over a channel that is in the feedback loop of an unstable scalar linear process, and
proposes the notion of anytime capacity as the appropriate figure of merit for such channels. In essense,
the encoder is causal and the probability of error in decoding a source symbol that was transmitted d
time instants ago should decay exponentially in the decoding delay d.
Although the connection between communication reliability and control is clear, very little is known
about error-correcting codes that can achieve such reliabilities. Prior to the work of [5], and in a different
context, [1] proved the existence of codes which under maximum likelihood decoding achieve such
reliabilities and referred to them as tree codes. Note that any real-time error correcting code is causal
and since it encodes the entire trajectory of a process, it has a natural tree structure to it. [1] proves the
existence of nonlinear tree codes and gives no explicit constructions and/or efficient decoding algorithms.
Much more recently [6] proposed efficient error correcting codes for unstable systems where the state
grows only polynomially large with time. So, for linear unstable systems that have an exponential growth
rate, all that is known in the way of error correction is the existence of tree codes which are, in general,
non-linear. When the state of an unstable scalar linear process is available at the encoder, [7] and
[8] develop encoding-decoding schemes that can stabilize such a process over the binary symmetric
channel and the binary erasure channel respectively. But when the state is available only through noisy
measurements, little is known in the way of stabilizing an unstable scalar linear process over a stochastic
communication channel.
The subject of error correcting codes for control is in its relative infancy, much as the subject of block
coding was after Shannon’s seminal work in [9]. So, a first step towards realizing practical encoder-
decoder pairs with anytime reliabilities is to explore linear encoding schemes. We consider rate R = k
n
causal linear codes which map a sequence of k-dimensional binary vectors b ∞ to a sequence of
τ τ=0
{ }
n dimensional binary vectors c ∞ where c is only a function of b t . Such a code is anytime
τ τ=0 t τ τ=0
− { } { }
reliable if at all times t and delays d d , P(cid:0)ˆb = b (cid:1) η2−βnd for some β > 0. We show that
o t−d|t t−d
≥ (cid:54) ≤
linear tree codes exist and further, that they exist with a high probability. For the binary erasure channel,
we propose a maximum likelihood decoder whose average complexity of decoding is constant per each
time iteration and for which the probability that the complexity at a given time t exceeds KC3 decays
exponentially in C. This allows one to stabilize a partially observed unstable scalar linear process over
a binary erasure channel and to the best of the authors’ knowledge, this has not been done before.
In Section II, we present some background and motivate the need for anytime reliability with a simple
example. In Section III, we come up with a sufficient condition for anytime reliability in terms of the
weight distribution of the code. In Section IV, we introduce the ensemble of time invariant codes and use
the results from Section III to prove that time invariant codes with anytime reliability exist with a high
probability. In Section V, we present a simple decoding algorithm for the BEC and present simulations
in Section VI to demonstrate the efficacy of the algorithm.
II. BACKGROUND AND PROBLEM SETUP
Owingtothedualitybetweenestimationandcontrol,theessentialcomplexityofstabilizinganunstable
process over a noisy communication channel can be captured by studying the open loop estimation of the
same process. So, we will illustrate the kind of communication reliability needed for control by analyzing
the open loop estimation of the following random walk.
A toy example: Consider estimating the following random walk, x = λx +w , where w = 1 w.p
t+1 t t t
±
1, x = 0 and λ > 1. Suppose an observer observes x and communicates over a noisy communication
2 0 | | t
channel to an estimator. The observer clearly needs to communicate one bit telling whether w is +1 or
t
11. Now the estimator’s estimate of the state, xˆ , is given by
t+1|t
−
t
(cid:88)
xˆ = λ wˆ (1)
t+1|t t−j j|t
j=0
Suppose Pe = P (cid:0)argmin (wˆ = wˆ ) = t d+1(cid:1), i.e., the position of the earliest erroneous wˆ is
d,t j j|t (cid:54) j − j|t
1If the channel is binary input, it should clearly allow at least one usage for each time step of the system evolution
at time j = t d+1. From (1), we can write E(cid:12)(cid:12)xt+1 xˆt+1|t(cid:12)(cid:12)2 as
− −
(cid:12) n (cid:12)2 (cid:12) t (cid:12)2
(cid:88) P (cid:0)w0:t,wˆ0:t|t(cid:1)(cid:12)(cid:12)(cid:88)λt−j(wj wˆj|t)(cid:12)(cid:12) = (cid:88)Pde,t(cid:12)(cid:12) (cid:88) λt−j(wj wˆj|t)(cid:12)(cid:12) λ (cid:88)Pde,t λ 2d
(cid:12) − (cid:12) (cid:12) − (cid:12) ≤ | | | |
w ,wˆ j=1 d≤t j=t−d+1 d≤t
0:t 0:t|t
Clearly, a sufficient condition for limsuptE(cid:12)(cid:12)xt+1 xˆt+1|t(cid:12)(cid:12)2 to be finite is as follows
−
Pe λ −2d−δ d d , t > t and δ > 0 (2)
d,t o o
≤ | | ∀ ≥
where d and t are constants that do no depend on t,d. In the above example, noise is discrete valued. If
o o
it is not discrete, for eg, a uniform random variable on a bounded interval, then apart from the reliability
criterion above, one would also need a minimum rate of communication that depends on the size of
log λ . But codes that communicate at a positive rate and satisfy the above reliability criterion have
2
| |
been more or less elusive. Consider the following unstable scalar linear process
Plant Observer
x = λx +u +w y = x +v
t+1 t t t t t t
C < ∞
u
t
Controller
Fig. 1: System model
x = λx +u +w , y = x +v (3)
t+1 t t t t t t
where λ > 1, u is the control input and, w < W and v < V are bounded process and measurement
| | t | t| 2 | t| 2
noisevariables.Themeasurements y aremadebyanobserverwhilethecontrolinputs u areapplied
t t
{ } { }
by a remote controller that is connected to the observer by a noisy communication channel. Naturally,
the measurements y will need to be encoded by the observer to provide protection from the noisy
0:t−1
channel while the controller will need to decode the channel output to estimate the state x and apply
t
a suitable control input u . This can be accomplished by employing a channel encoder at the observer
t
and a decoder at the controller. For simplicity, we will assume that the channel input alphabet is binary.
b1 E c1 = f1(b1) C z1 D ˆb11
|
b2 N c2 = f2(b1,b2) H z2 E ˆb12,ˆb22
| |
C A C
. O . N . O .
. . . .
. . . .
D N D
E E E
bt R ct = ft(b1,...,bt) L zt R ˆb1t,...,ˆbtt
| |
Fig. 2: Causal encoding and decoding
Suppose one time step of system evolution in (3) corresponds to n channel uses2. Then, at each instant
of time t, the operations performed by the observer, the channel encoder, the channel decoder and the
controller can be described as follows. The observer generates a k bit message, b 0,1 k, that is a
t
− ∈ { }
causal function of the measurements, i.e., it depends only on y . Then the channel encoder causally
0:t
encodes b 0,1 kt to generate the n channel inputs c 0,1 n. Note that the rate of the channel
0:t t
∈ { } ∈ { }
encoder is R = k/n. Denote the n channel outputs corresponding to c by z n, where denotes the
t t
∈ Z Z
channel output alphabet. Using the channel outputs received so far, i.e., z nt, the channel decoder
0:t
∈ Z
generates estimates ˆb of b , which, in turn, the controller uses to generate the control input
τ|t τ≤t τ τ≤t
{ } { }
u . This is illustrated in Fig. 2. Note that we do not assume any channel feedback. Then using the
t+1
lattice quantizer argument presented in [5], in the limit of large n and large λ, we have the following
sufficient condition on the performance of the encoder-decoder pair so that the unstable process in (3)
can be stabilized
Lemma 2.1 (Theorem 5.2 [5]): It is possible to control the unstable scalar process (3) over a noisy
communication channel so that limsup E x m < if, for some rate R > 1 log λ and exponent
t | t| ∞ n 2| |
β > m log λ , we have
n 2| |
(cid:16) (cid:17)
P min τ :ˆb = b = t d+1 η2−βnd, d d , t > t
τ|t τ o o
{ (cid:54) } − ≤ ∀ ≥
where d and t are constants independent of d,t.
o o
In what follows, we will demonstrate causal linear codes which under maximum likelihood decoding
achieve such exponential reliabilities.
2In practice, the system evolution in (3) is obtained by discretizing a continuous time differential equation. So, the interval
of discretization could be adjusted to correspond to an integer number of channel uses, provided the channel use instances are
close enough.
III. LINEAR ANYTIME CODES - A SUFFICIENT CONDITION
As discussed earlier, a first step towards developing practical encoding and decoding schemes for
automatic control is to study the existence of linear codes with anytime reliability. We will begin by
defining a causal linear code.
Definition 1 (Causal Linear Code): A causal linear code is a sequence of linear maps f : 0,1 kτ
τ
{ } (cid:55)→
0,1 n and hence can be represented as
{ }
f (b ) = G b +G b +...+G b (4)
τ 1:τ τ1 1 τ2 2 ττ τ
where G 0,1 n×k
ij
∈ { }
We denote c (cid:44) f (b ). Note that a tree code is a more general construction where f need not be
τ τ 1:τ τ
linear. Also note that the associated code rate is R = k. One can alternately represent a causal linear code
n
by an infinite dimensional block lower triangular generator matrix G or equivalently as an infinite
n,R
dimensional block lower triangular parity check matrix, H .
n,R
   
G 0 ... ... ... H 0 ... ... ...
11 11
   
 G G 0 ... ...   H H 0 ... ... 
 21 22   21 22 
Gn,R =  ... ... ... ... ... , Hn,R =  ... ... ... ... ...  (5)
   
   
 Gτ1 Gτ2 ... Gττ 0   Hτ1 Hτ2 ... Hττ 0 
 ... ... ... ... ...   ... ... ... ... ... 
where H 0,1 n×n and n = n(1 R)3. In fact, we present all our results in terms of the parity
ij
∈ { } −
check matrix. Before proceeding further, it is useful to introduce some notation
A. Notation
1) Ht (cid:44) nt nt leading principal minor of H
n,R × n,R
(cid:110) (cid:111)
2) (cid:44) c 0,1 nt : Ht c = 0
Ct ∈ { } n,R
3) (cid:44) c : c = 0, c = 0
t,d t τ<t−d+1 t−d+1
C { ∈ C (cid:54) }
4) Nt (cid:44) c : c = w
w,d |{ ∈ Ct,d (cid:107) (cid:107) }|
5) wt (cid:44) argmin (Nt = 0)
min,d w w,d (cid:54)
(cid:16) (cid:17)
6) Pe (cid:44) P min τ :ˆb = b = t d+1
t,d { τ|t (cid:54) τ} −
where c denotes the Hamming weight of c.
(cid:107) (cid:107)
3While for a given generator matrix, the parity check matrix is not unique, when G is block lower, it is easy to see that
n,R
H can also be chosen to be block lower.
n,R
B. A Sufficient Condition
The objective is to study the existence of causal linear codes which under ML decoding guarantee
Pe η2−βd, t, d d (6)
d,t o
≤ ∀ ≥
where d is a constant independent of d,t. In what follows, we will develop a sufficient condition for a
o
linear code to be anytime reliable in terms of its weight distribution. Suppose the decoding instant is t
and without loss of generality, assume that the all zero codeword is transmitted, i.e., c = 0 for τ t.
τ
≤
We are interested in the error event where the earliest error in estimating b happens at τ = t d+1,
τ
−
i.e., ˆb = 0 for all τ < t d+1 and ˆb = 0. Note that this is equivalent to the ML codeword, cˆ,
τ|t t−d+1|t
− (cid:54)
satisfying cˆ = 0 and cˆ = 0, and Ht having full rank so that cˆ can be uniquely mapped
τ<t−d+1 t−d+1 (cid:54) n,R
to a transmitted sequence ˆb. Then we have
 
(cid:91) (cid:88)
Pe = P  0 is decoded as c P (0 is decoded as c) (7a)
t,d
≤
c∈C c∈C
t,d t,d
Now,itiswellknown(foreg,see[10])that,undermaximumlikelihooddecoding,P (0 is decoded as c)
≤
ζ(cid:107)c(cid:107), where ζ is the Bhattacharya parameter, i.e.,
∞
(cid:90)
(cid:112)
ζ = p(z X = 1)p(z X = 0)dz
| |
−∞
where, z and X denote the channel output and input respectively. From (7a), it follows that
(cid:88)
Pe Nt ζw
t,d w,d
≤
wt ≤w≤nd
min,d
If wt αnd and Nt 2θw for some θ < log (1/ζ), then
min,d ≥ w,d ≤ 2
Pte,d η2−αnd(log2(1/ζ)−θ) (8)
≤
where η = (1 2log2(1/ζ)−θ)−1. So, an obvious sufficient condition for Hn,R can be described in terms
−
of wt and Nt as follows. For some θ < log (1/ζ), we need
min,d w,d 2
wt αnd, Nt 2θw t, d d (9a)
min,d w,d o
≥ ≤ ∀ ≥
where d is a constant that is independent of d,t. This brings us to the following definition
o
Definition 2 (Anytime distance and Anytime reliability): WesaythatacodeH has(α,θ,d ) anytime
n,R o
−
distance, if the following hold
1) Ht is full rank for all t > 0
n,R
2) wt αnd, Nt 2θw for all t > 0 and d d .
min,d ≥ w,d ≤ ≥ o
Also, we say that a code H is (R,β,d ) anytime reliable if, under ML decoding
n,R o
−
Pe η2−βnd, t > 0, d d (9b)
t,d o
≤ ∀ ≥
where, d is a constant independent of t,d.
o
IV. LINEAR ANYTIME CODES - EXISTENCE
We will begin by proving the existence of such codes over a finite time horizon, T, i.e., Pe
d,t ≤
η2−βd, t T, d d . We will then prove their existence for all time.
o
∀ ≤ ≥
A. Finite Time Horizon
Over a finite time horizon, T, a causal linear code is represented by a block lower triangular parity
check matrix H 0,1 nT×nT. The following Theorem guarantees the existence of a H such
n,R,T n,R,T
∈ { }
that (9) is true for all t T.
≤
Theorem 4.1 (Appropriate Weight Distribution): For each time T > 0, rate R > 0, α < H−1(1 R)
−
and θ > log (1/(21−R 1)), there exists a causal linear code H(n,k,T) that has (α,θ,d ) anytime
2 o
− −
distance, where d is a constant independent of d, t and T.
o
H−1(1 R) is the smaller root of the equation H(x) = 1 R, where H(.) is the binary entropy function.
− −
The proof is by induction and is detailed in the Appendix. Theorem 4.1 proves the existence of finite
dimensional causal linear codes, H , that are anytime reliable for decoding instants upto time T.
n,R,T
In the following subsection, we demonstrate the existence of infinite dimensional causal linear codes,
H , that are anytime reliable for all decoding instants. We also show that such codes drawn from an
n,R
appropriate ensemble are anytime reliable with a high probability.
B. Time Invariant Codes
Consider causal linear codes with the following Toeplitz structure
 
H 0 ... ... ...
1
 
 H H 0 ... ... 
 2 1 
HTZ =  ... ... ... ... ... 
n,R  
 
 
 Hτ Hτ−1 ... H1 0 
 ... ... ... ... ... 
The superscript TZ in HTZ denotes ‘Toeplitz’. HTZ is obtained from H in (5) by setting H =
n,R n,R n,R ij
H for i j. Due to the Toeplitz structure, we have the following invariance, wt = wt(cid:48) and
i−j+1 ≥ min,d min,d
Nt = Nt(cid:48) for all t,t(cid:48). The code HTZ will be referred to as a time-invariant code. The notion of time
w,d w,d n,R
invariance is analogous to the convolutional structure used to show the existence of infinite tree codes in
[1]. This time invariance allows one to prove that such codes which are anytime reliable are abundant.
Definition 3 (The ensemble TZ ): The ensemble TZ of time-invariant codes, HTZ, is obtained as
p p n,R
follows, H is any full rank binary matrix and for τ 2, the entries of H are chosen i.i.d according to
1 τ
≥
Bernoulli(p), i.e., each entry is 1 with probability p and 0 otherwise.
For the ensemble TZ , we have the following result
p
Theorem 4.2 (Abundance of time-invariant codes): ForeachR > 0,α < H−1[(1 R)log (1/(1 p))]
2
− −
and θ > log (cid:2)(1 p)−(1−R) 1(cid:3), we have
2
− − −
P (cid:0)HTn,ZR has (α,θ,do) anytime distance(cid:1) 1 2−Ω(ndo) (10)
− ≥ −
Wecannowusethisresulttodemonstrateanachievableregionofrate-exponentpairsforagivenchannel,
i.e., the set of rates R and exponents β such that one can guarantee (R,β) anytime reliability using linear
codes. To determine the values of R that will satisfy (8), note that we need
log (1/(21−R 1)) < log (1/ζ) = R < 1 log (1+ζ)
2 2 2
− ⇒ −
With this observation, we have the following Corollary.
Corollary 4.3: For any rate R and exponent β such that
(cid:18) (cid:18) (cid:19) (cid:19)
R < 1 log2(1+ζ) and β < H−1(1 R) log 1 log (cid:2)(1 p)−(1−R) 1(cid:3) (11a)
2 2
− log (1/(1 p)) − ζ − − −
2
−
if HTZ is chosen from TZ , then
n,R p
P (cid:0)HTn,ZR is (R,β,do) anytime reliable(cid:1) 1 2−Ω(ndo)
− ≥ −
Note that by choosing p small, we can trade off better rates and exponents with sparser parity check
(cid:112)
matrices. Note that for BEC((cid:15)), ζ = (cid:15) and for BSC((cid:15)), ζ = 2 (cid:15)(1 (cid:15)). For BSC((cid:15)) with p = 1, the
− 2
threshold for rate in Corollary 4.3 becomes R < 1 2log (√(cid:15)+√1 (cid:15)). It turns out that this can be
2
− −
strengthened as follows.
Theorem 4.4 (Tighter bounds for BSC((cid:15))): For any rate R and exponent β such that
R < 1 H((cid:15)), β < KL(cid:0)H−1(1 R) min (cid:15),1 (cid:15) (cid:1)
− − (cid:107) { − }
if HTZ is chosen from TZ , then
n,R 1
2
P (cid:0)HTn,ZR is (R,β,do) anytime reliable(cid:1) 1 2−Ω(ndo)
− ≥ −
C. Stabilizable Region
Using the thresholds obtained in Corollary 4.3 and Theorem 4.4, we can discuss the range of λ for
| |
which the mth moment of x in (3) can be stabilized over some common channels. Using Lemma 2.1,
t
an anytime reliable code with rate R and exponent β can stabilize the process in (3) for all λ such that
nβ
log λ < min nR,
2
| | { m }
So, a scalar unstable linear process in (3) can be stabilized over a channel with Bhattacharya parameter
ζ provided
βn
log λ < log λ = sup min nR, (12)
2 2 max
| | | | { m }
R<R ,β<β
ζ ζ,R
(cid:16) (cid:16) (cid:17) (cid:17)
where R = 1 log (1+ζ) and β = H−1(1 R) log 1 +log (cid:0)21−R 1(cid:1) are obtained by
ζ − 2 ζ,R − 2 ζ 2 −
setting p = 1 in (11). For the BSC((cid:15)), using Theorem 4.4, one can tighten (12) by replacing R with
2 ζ
1 H((cid:15)) and β with KL(cid:0)H−1(1 R) min (cid:15),1 (cid:15) (cid:1). For m = 2, the stabilizable region for the
ζ,R
− − (cid:107) { − }
BEC and BSC is shown in Fig 3 where λ 1 is plotted against the channel parameter.
max n
| |
V. DECODING OVER THE BEC
Owing to the simplicity of the erasure channel, it is possible to come up with an efficient way to
perform maximum likelihood decoding at each time step. We will show that the average complexity of