Generalized Approximate Message-Passing Decoder for Universal Sparse Superposition Codes

1.1 MB·

by  Erdem Biyik

Generalized Approximate Message-Passing Decoder
for Universal Sparse Superposition Codes
Erdem Bıyık∗, Jean Barbier† and Mohamad Dia†
Bilkent University, Ankara, Turkey
∗
Communication Theory Laboratory, EPFL, Lausanne, Switzerland
†
[email protected], {jean.barbier, mohamad.dia}@epfl.ch
Abstract—Sparse superposition (SS) codes were originally this work we fill this gap by studying the GAMP decoder
proposed as a capacity-achieving communication scheme over for SS codes over various memoryless channels. We focus
the additive white Gaussian noise channel (AWGNC) [1]. Very
on the AWGNC (for completeness with previous studies [11,
recently, it was discovered that these codes are universal, in the
12]),binaryerasurechannel(BEC),Zchannel(ZC)andbinary
sense that they achieve capacity over any memoryless channel
undergeneralizedapproximatemessage-passing(GAMP)decod- symmetric channel (BSC). However, the present decoder and
ing [2], although this decoder has never been stated for SS analysis remain valid for any memoryless channel.
7 codes. In this contribution we introduce the GAMP decoder OurexperimentsconfirmthatSErecursionof[2]accurately
1 for SS codes, we confirm empirically the universality of this tracksGAMP.Usingthepotentialofthecodewealsocompare
0 communicationschemethroughitsstudyonvariouschannelsand
the performance of GAMP to the optimal MMSE decoder. In
2 weprovidethemainanalysistools:stateevolutionandpotential.
We also compare the performance of GAMP with the Bayes- addition, our empirical study confirms the asymptotic results
n
optimal MMSE decoder. We empirically illustrate that despite of [2]: the performance of SS codes under GAMP decoding
a
the presence of a phase transition preventing GAMP to reach can be significantly increased towards capacity using spatial
J
the optimal performance, spatial coupling allows to boost the
coupling,asalreadyobservedfortheAWGNC[12].Moreover,
3 performance that eventually tends to capacity in a proper limit.
1 We also prove that, in contrast with the AWGNC case, SS codes we prove that for binary input channels, SS codes have a
forbinaryinputchannelshaveavanishingerrorfloorinthelimit vanishingerrorfloorinthelimitoflargecodewordsevenwith

] of large codewords. Moreover, the performance of Hadamard- finite sparsity. This means that when decoding is possible,
T
based encoders is assessed for practical implementations. optimal decoding is asymptotically perfect as well as GAMP
I
. decodinguntilsomethreshold,averypromisingfeaturewhich
s I. INTRODUCTION
c is not present for the real-valued input AWGNC. Keeping in
[ Sparse superposition codes were introduced by Barron and mindpracticality,wefocusourempiricalstudyonHadamard-

1 JosephforcommunicationovertheAWGNC[1].Thesecodes based coding operators that allow to drastically reduce the
v were proven to achieve the Shannon capacity using power encoding and decoding complexity, while maintaining good
0 allocation and various efficient decoders [3,4]. A decoder performance for moderate block-lengths [11].
9
based on approximate message-passing (AMP), originally
5 developed for compressed sensing [5,6], was introduced in II. SPARSESUPERPOSITIONCODES:SETTING
3
0 [7]. The authors in [8] proved, using the state evolution (SE) In SS codes, the message x=[x1,...,xL] is a vector made
1. analysis [9,10], that AMP allows to achieve capacity using of L B-dimensional sections. Each section xl, l∈{1,...,L},
powerallocation.Atthesametime,spatiallycoupledSScodes satisfies a hard constraint: it has a single non-zero component
0
7 wereintroducedin[11,12]andempiricallyshowntoapproach equals to 1 whose position encodes the symbol to transmit. B
1 capacity under AMP without power allocation and to perform is the section size (or alphabet size) and we set N:=LB. For
: much better than power allocated ones. Recently, AMP for the theoretical analysis we consider random codes generated
v
i spatially coupled SS codes was shown to saturate the so- by a coding matrix A RM×N drawn from the ensemble
X ∈
calledpotentialthreshold,relatedtotheBayes-optimalMMSE of Gaussian matrices with i.i.d entries (0,σ2). For the
r performance, which tends to capacity in a proper limit [13]. practical implementation, fast Hadamard∼-bNased opAerators are
a
This set of works combined with the excellent performance used instead as they exhibit very good performances. Despite
of SS codes over the AWGNC motivated their study for any thelackofrigorintheanalysisforsuchoperators,theyremain
memoryless channel under GAMP decoding [2]. GAMP was good predictive tools [11]. The codeword is Ax RM. We
∈
introducedasageneralizationofAMPforgeneralizedestima- enforcethepowerconstraint Ax 2/M=1bytuningσ2.The
|| ||2 A
tion[14].In[2]theauthorsshowedthat,undertheassumption cardinality of the code is BL. Hence, the (design) rate is R=
that SE [10] tracks GAMP for SS codes, spatially coupled SS Llog (B)/M and the code is thus specified by (M,R,B).
2
codes achieve the capacity of any memoryless channel under The aim is to communicate through a known memo-
GAMP decoding. However, GAMP has never been explicitly ryless channel W. This requires to map the continuous-
stated or tested as a decoder for SS codes other than for the valued codeword onto the input alphabet of W. The con-
AWGNC, in which case GAMP and AMP are identical. In catenation of this mapping operation and the channel it-
100
Algorithm 1 GAMP(y,A,B,t ,u)
max
1: (cid:98)x(0) =0N,1, τx(0) =(1/B)1N,1
2: s( 1) =0M,1, t=0, e(0) = (cid:46) Initializations 10-1
9876543::::::: w−hirτsτpτle(((r(s(p(tttttt))t))))≤t======maxA(cid:98)Ax−g(a(o((cid:98)◦xg(tun2)τt(o(cid:48)d(tτu+)ps(ttx((e(−)tptτ()))t((cid:124)r(,)τttA)y)≥p(,,t◦◦y)τ2,u◦()p(τ∞s(cid:124)ts)d(cid:124)()p((t)t◦o)t)−−A)(cid:46)11))O(cid:124)u(cid:46)tp(cid:46)OutIuntnppouuntt-llliiinnneeeaaarrr sssttteeeppp ESM1100--320 RRRRRR======000000::::::664444,,44,,,,GSGS2GSttAAaatAtatMMeetMePPEEPEvvDDovoDeeloluucceluoottciiotddooidNeenonrr4enrumbero6fIteration8s 10 12
1110:: (cid:98)xτ(x(tt++11)) == gτir(nt()r◦(t)g,i(cid:48)nτ(r(rt()t)),τr(t))(cid:46) Input non-linear step FLinisgt=.an1c:2eS1s1E),otBrvaecrk=itnhge4tBhaEendCGAGwMaituhPssediareancsoucdroeedrpi(nragovbeaorbapgieleritadytoo(cid:15)rvse.=rM100.o10ntareanndcdaofromlor
1132:: te(t) == t||(cid:98)+x(t1+1)−(cid:98)x(t)||22/L iTnhteegaralgtioornithwmitihc2th×re1s0h4olsdamRpGlAesMiPs us0e.d55forantdhethceomgrpeuetnaticounrvoefsSaEre.
≈
14: return (cid:98)x(t) (cid:46) The prediction scores for each bit forarateaboveit:decodingfails.Incontrast,theblueandredcurves
arebelowRGAMP:decodingsucceeds.Afterthelastpointsofthese
curves, both SE and the GAMP curves fall to 0 MSE.
self can be interpreted as an effective memoryless channel Moreover, the estimate of the posterior variance, which quan-
P (yAx)=(cid:81)M P (y [Ax] ). For the channels we focus tifieshow“confident”GAMPisinitscurrentestimate,equals
onou,tPo|ut(yµ|[Axµ]µ=)1isoeutxprµe|ssedµas follows: cτom◦pongei(cid:48)nn(trw,iτse) p=artiEa[lXd◦e2r|iRvati=verw].−r.tgiitns(rfi,rτst)◦a2rgu(gmi(cid:48)nenits, atnhde
AWGNC: (y [Ax] ,1/snr),
µ µ
• N | similarly for g ). Plugging P yields the componentwise
BEC: (1 (cid:15))δ(y sign([Ax] ))+(cid:15)δ(y ), o(cid:48)ut 0
µ µ µ
• − − expression of the denoiser and the variance term:
BSC: (1 (cid:15))δ(y sign([Ax] ))+(cid:15)δ(y +sign([Ax] )),
µ µ µ µ
•• δZ(Csi:gδn((s[Ai−gxn](µ[A) x]1µ−))δ+(y1µ)((cid:15)1δ)(,yµ−1)+(1−(cid:15))δ(yµ+1))+ (cid:40)[gin(r,τ)]i = (cid:80)j∈exlipe(x(2pr(i(−2r1j)−/(12)τ/i()2)τj)),
− − [τ g (r,τ)] =[g (r,τ)] (1 [g (r,τ)] ),
where snr is the signal-to-noise of the AWGNC, (cid:15) the erasure ◦ i(cid:48)n i in i − in i
or flip probability of the BEC, ZC and BSC. The sign maps l being the section to which belong the ith scalar component.
i
the Gaussian distributed codeword components onto the input In contrast with g that only depends on P , g depends
in 0 out
alphabets of the binary input channels. on the communication channel and acts componentwise. Its
Note that for the asymmetric ZC, the symmetric map generalformandspecificexpressionsforthestudiedchannels
sign([Ax]µ) leads to a sub-optimal uniform input distribution. are given in Table I along with the necessary derivatives.
The symmetric capacity of the ZC differs from Shannon’s The complexity of GAMP is dominated by the (MN)=
O
capacity but the difference is small, and similarly for the (L2Bln(B)) matrix-vector multiplications. In terms of
O
algorithmicthreshold,see[2].Wethusconsiderthissymmetric memory, it is necessary to store A which can be problematic
setting for the sake of simplicity. The other channels are sym- forlargecodes.FastHadamard-basedoperatorsconstructedas
metric, this map thus leads to the optimal input distribution. in[11],withrandomsub-sampledmodesofthefullHadamard
operator, allow to achieve a lower (Lln(B)ln(BL)) decod-
III. THEGAMPDECODER O
ingcomplexityandstronglyreducethememoryneed[12,15].
WeconsideraBayesiansettingandassociatetothemessage
the posterior P(xy,A)=P (yAx)P (x)/P(yA). The hard IV. STATEEVOLUTIONANDTHEPOTENTIAL
out 0
| | |
constraints for the sections are enforced by the prior P0(x)= We now present the analysis tools of the L perfor-
(cid:81)Ll=1p0(xl) with p0(xl) = B−1(cid:80)i lδxi,1(cid:81)j l,j=iδxj,0, mance of SS codes under GAMP and MMSE de→cod∞ing when
where i l are the B scalar comp∈onents ind∈ices(cid:54) of the Gaussian matrices are used: state evolution and potential.
{ ∈ }
section l. The GAMP decoder aims at performing MMSE es-
A. State evolution
timationbyapproximatingtheposteriormeanofeachsection.
In the GAMP decoder Algorithm 1, denotes element- The asymptotic performance of GAMP with Gaussian i.i.d
◦
wise operations. GAMP was originally derived for scalar coding matrices is tracked by SE, a scalar recursion [2,9,
estimation. In this generalization to the vectorial setting of 10,14] analogous to density evolution for low-density parity-
SS codes, whose derivation is similar to the one of AMP for checkcodes.NotethatalthoughSEisnotrigorousforvectorial
SS codes found in [12], only the input non-linear steps differ setting, the rigorous analysis of [8] and the present empirical
from canonical GAMP [14]: here the so-called denoiser g results strongly suggest that it is exact, which we conjecture.
in
acts sectionwise instead of componentwise. In full generality, The aim is to compute the asymptotic MSE of the GAMP
ivtariisabdleefinRed=asX[+14Z(cid:98)] gwini(thr,τX):=PE0[Xan|Rd=Z(cid:98)r] for(0th,ediraagn(dτo)m). iesstiemquatievaEle(ntt):t=olirmecLu→rs∞ive(cid:107)l(cid:98)yx(tc)o−mxp(cid:107)u22t/eLt.hIet tMurMnsSoEutTt(hEat)th:=is
∼ ∼ N
TABLE I: The expressions for gout, −go(cid:48)ut and F.
[gout(p,y,τ)]i [−go(cid:48)ut(p,y,τ)]i F(p|E)
General (E[Zi|pi,yi,τi]−pi)/τi (τi−Var[Zi|pi,yi,τi])/τi2 See(1)
Yi∼Pout(·|zi),Zi∼N(pi,τi) Yi∼Pout(·|zi),Zi∼N(pi,τi)
AWGNC yi−pi 1 1
τi+1/snr τi+1/snr 1/snr+E
BEC (pi−ki)h+i+(pZi+BEkCiτ)hi−i+2(cid:15)δ(yi)pi−pτii τ1i−(p2i+τi−ki(cid:48))h+i+(p2i+ZτBiE+Ckτi(cid:48)i)2h−i+2(cid:15)δ(yi)(τi+p2i)+(cid:0)[gout(p,y,τ)]i+pτii(cid:1)2 QQ(cid:48)(21(−1−Q(cid:15)))
ZC (pi−ki)vi++Z(ZpCiτ+iki)δ(yi−1)−pτii τ1i−(p2i+τi−ki(cid:48))vi+Z+Z(Cpτ2i+i2τi+ki(cid:48))δ(yi−1)+(cid:0)[gout(p,y,τ)]i+pτii(cid:1)2 QQ+(cid:48)2(cid:15)((11−−(cid:15)Q)2)+Q(cid:48)12−(Q1−(cid:15))
BSC (pi−ki)vZi+B+SC(τpii+ki)vi−−pτii τ1i−(p2i+τi−ki(cid:48))ZvBi+S+C(τpi2i2+τi+ki(cid:48))vi−+(cid:0)[gout(p,y,τ)]i+pτii(cid:1)2 (Q+(cid:15)−2Q(cid:15)Q(cid:48)2)((11−−2Q(cid:15))−2(cid:15)+2(cid:15)Q)
h+i =(1−(cid:15))δ(yi+1), h−i =(1−(cid:15))δ(yi−1), vi+=(1−(cid:15))δ(yi+1)+(cid:15)δ(yi−1), vi−=(1−(cid:15))δ(yi−1)+(cid:15)δ(yi+1),
ki=exp(cid:0)−2τpi2i(cid:1)(cid:112)2τi/π+erf(cid:0)√p2iτi(cid:1)pi, ki(cid:48)=kipi+erf(cid:0)√p2iτi(cid:1)τi, Q=12erfc(√−2pE), Q(cid:48)=exp(cid:0)−2Ep2(cid:1)(cid:14)√2πE
ZBEC=erfc(cid:0)√p2iτi(cid:1)h+i+(cid:0)1+erf(cid:0)√p2iτi(cid:1)(cid:1)h−i+2(cid:15)δ(yi),ZZC=erfc(cid:0)√p2iτi(cid:1)vi++(cid:0)1+erf(cid:0)√p2iτi(cid:1)(cid:1)δ(yi−1),ZBSC=erfc(cid:0)√p2iτi(cid:1)vi++(cid:0)1+erf(cid:0)√p2iτi(cid:1)(cid:1)vi−
E [ S E[XS+(Σ(E)/b)Z] 2] of a single section (S p ) sections, spatial coupling may allow to boost the performance
S,Z (cid:107) − | (cid:107)2 ∼ 0
sent through an equivalent AWGNC (Z (0,I )) of noise oftheschemebyincreasingtheGAMPalgorithmicthreshold.
B
∼N
variance (Σ(E)/b)2, b2:=log (B). This formulation is valid
2
for any memoryless channel [2], P being reflected in B. Potential formulation
out
Σ(E) :=√R[(cid:82) dp (p0,1 E) (pE)] 1/2, The SE (2) is associated with a potential Fu(E), whose
 (cid:82) N | − F | − stationary points correspond to the fixed points of SE:
(pE) := dyf(y p,E)(∂ lnf(y x,E))2 , (1)
fF(y|p,E) :=(cid:82) dzP |(y z) (xz p,E).| x=p ∂EFu(E)|E0=0⇔T(E0)=E0. For SS codes it is [2]:
| out | N | 
F (E) :=U (E) S (Σ(E))
F istheFisherinformationofpassociatedwithf,seeTableI. Uu(E) := u −E u 1E [(cid:82) dyφlog (φ)],
TZhCeapndintBegSrCa.liDneΣfinceanbenumericallycomputedfortheBEC, Suu(Σ(E)) :=−EZ2(cid:2)lnlo(2g)BΣ((cid:0)E1)2+−(cid:80)RBi=Z2ei(Z,Σ(E)2/b)(cid:1)(cid:3),

ggii((nn12))((ΣΣ,,zz)) ::==(cid:2)(cid:2)11++ee−Σb22Σb+22((cid:80)z1−Bj=z22)eΣbΣb+(z(cid:80)j−Bkz=1)3(cid:3)e−(1zk,−z2)Σb(cid:3)−1. Nwh(e0r,e1)φa=nφd(eyi|(ZZ,,Ex))::==e(cid:82)xdps(cid:0)P(oZuit(−y|Zs1)N)/x(s−|Z1√/x12−(cid:1).E,E), Z∼
It has been recently shown for random linear estimation,
TheMMSEoftheequivalentAWGNCisobtainedaftersimple including compressed sensing and SS codes with AWGN
algebra [12] and reads [16,17], that min F (E) equals the asymptotic mutual
E [0,1] u
∈
T(E)=E [(g(1)(Σ(E),Z) 1)2+(B 1)g(2)(Σ(E),Z)2]. information (up to a trivial additive term) and that E(cid:101) :=
Z in − − in argmin F (E) equals the asymptotic MMSE. A proof
E [0,1] u
Hereg(1) isinterpretedastheposteriormeanapproximatedby for all m∈emoryless channels remains to be done, but we
in
GAMP of the non-zero component in the transmitted section conjecture that it remains true under mild conditions on P .
out
while g(2) corresponds to the remaining components. The SE Using these properties of the potential and its link with SE,
in
recursion tracking the MSE of GAMP is then it is possible to assess the performances of the GAMP and
MMSE decoders by looking at its minima. GAMP decoding
E =T(E ), t 0, (2)
(t+1) (t) ≥ is possible (and asymptotically optimal as it reaches the
initialized with E =1. Hence, the asymptotic MSE reached MMSE E(cid:101), black dot in Fig. 2) for rates lower or equal
(0)
by GAMP upon convergence is E . Moreover, define the to R , whose equivalent definition is the smallest
( ) GAMP
asymptotic L error floor of S∞S codes E as the fixed solution of ∂F /∂E = ∂2F /∂E2 = 0; in other words it
u u
→∞ ∗
pointofSE(2)initializedfromE =0.Fig.1showsthatSE is the smallest rate at which a horizontal inflection point
(0)
properlytracksGAMPontheBEC.Notethatthesectionerror appears in the potential, see blue and red curves in Fig. 2.
rate(SER)ofGAMP,thefractionofwronglydecodedsections For R ]R ,R [, referred to as the hard phase, the
GAMP pot
∈
after hard thresholding of (cid:98)x(t), can also be asymptotically potential possesses another local min. (red dot) and the
tracked thanks to SE through a simple one-to-one mapping corresponding “bad” fixed point of SE prevents GAMP to
between E and the asymptotic SER at t [7,12]. reach E(cid:101); decoding fails (yellow curves). Finally, the rate at
(t)
UnderGAMPdecodingSScodesexhibit,asL ,asharp which the local and global min. switch roles is the potential
→∞
phase transition at an algorithmic threshold R below threshold R (purple curves). Optimal decoding is possible
GAMP pot
Shannon’scapacity.R isdefinedasthehighestratesuch as long as R < R as the MMSE switches at R from
GAMP pot pot
that for R R , (2) has a unique fixed point E =E a “low” to a “high” value. At higher rates GAMP is again
GAMP ( )
≤ ∞ ∗
(see[2]forformaldefinitions).InthisregimeGAMPdecodes optimal but leads to poor results as decoding is impossible.
well,seeredandbluecurvesofFig.1.IfR>RGAMP GAMP Note that if R<Rpot, then E =E(cid:101).
∗
decoding fails, see green curve. As we will see in the next
The proof of E =0, i.e the existence of the trivial fixed
∗
pointT(0)=0of(2),doesnotguaranteethatthisistheglobal
minimum of the potential in the hard phase; i.e it is a priori
possiblethatE =E(cid:101).Nevertheless,ourcarefulnumericalwork
∗(cid:54)
indicates that there exist at most two fixed points of SE at the
sametimeorequivalentlytwominimainthepotential,namely
E =E(cid:101)=E( ) if R ]RGAMP,Rpot[ or E =E(cid:101)=E( ) if
∗ (cid:54) ∞ ∈ ∗(cid:54) ∞
R>R (at least for the studied cases), see Fig. 2. This also
pot
agrees with the B analysis of the potential [2,12].
→∞
Let us now prove that E = 0 for the BEC, the proof
∗
for other binary input channels being similar. It starts by
noticing, from the definition of T(E) as the MMSE of
an AWGNC with noise parameter Σ(E), that a sufficient
condition for T(0) = 0 is lim Σ(E) = 0; indeed no
E 0
→
noise implies vanishing MMSE. From (1) this condition is
equivalent to limE 0IR(E)= that we now prove, where
I (E):=(cid:82) dp (p→0,1 E) (∞pE). Consider instead I (E)
wAhere :=A[E √NE,|E+√−E].FUsi|ngTableIfortheexpreEssion
E −
of (pE) for the BEC, this restricted integral is
F |
Fig. 2: Potential for the AWGNC with snr=100 (top) and the BEC
with (cid:15)=0.1 (bottom), in both cases with B =2. The MMSE is (1 (cid:15))(2π)−3/2 (cid:90) e−2(1p−2E)−pE2
the argminFu(E) (black dot). When the min. is unique (i.e R< IE(E)= −E√1 E dpQ(p,E)(1 Q(p,E)).
RGAMP,bluecurve)oriftheglobalmin.istherightmostone(R> − E −
Rpot, green curve), GAMP is asymptotically optimal, despite that Here Q(p,E) [CE,1 CE], with limE 0CE >0 for p
if R>Rpot it leads to poor results. The red dot is the local min., , E 1. Thi∈s implies−that K(E) := m→axp Q(p,E)(1∈
preventing GAMP to decode if R∈]RGAMP,Rpot[ (yellow curve). QE(p,E≤))= (1). Since the interval is of siz∈eE2√E, then−
O E
In the hard phase, where two minima coexist, spatial (1 (cid:15))(2π)−3/22√E (E+√E)2 (E+√E)2
coupling enables decoding [11] by “effectively suppressing” IE(E)≥ −E√1 EK(E) e− 2(1−E) − E . (3)
the spurious local min. of the potential. It implies that the −
tahlgeohriitghhmesict athttraeisnhaobllde oraftespuastiinalglycocuopulpeldedcoSdSescoudnedserRGGcAAMMPP, FIEro(Em)t<hisIRw(Ee)caans aFss(epr|tEt)h≥at0lim(reEc→al0lIitE(iEs a)=Fi∞she.rMinofroeromvae-r
decoding [2], saturates the potential threshold Rpot in the tion) and thus limE→0IR(E)=∞ which ends the proof.
FortheBSCandZCtheproofissimilar,themainingredient
limit of infinite coupled chains. This phenomenon is referred
being the squared Gaussian Q2 at the numerator of (pE),
to as threshold saturation and is understood as the generic (cid:48) F |
see Table I, which leads to similar expressions as (3) and thus
mechanism behind the excellent performances of coupled
the 1/√E divergence when E 0. We believe that the same
codes[2,18].Moreover,averyinterestingfeatureofSScodes →
mechanism holds for any binary input memoryless channel,
is that R itself approaches the capacity as B [2].
pot
→ ∞ implying a vanishing error floor as well as asymptotic perfect
These phenomena imply together that in these limits (infinite
decoding of GAMP below the algorithmic threshold.
chainlengthandB),spatiallycoupledSScodesunderGAMP
decoding are universal in the sense that they achieve the V. NUMERICALEXPERIMENTS
Shannon capacity of all memoryless channels.
InFig.3wecomparetheoptimalandGAMPperformances
in terms of attainable rate, denoted by R and R
pot GAMP
C. Vanishingerrorfloorforbinaryinputmemorylesschannels
respectively. For all channels, there exists, as long as the
Another promising feature of SS codes is related to their noise is not “too high”, a hard phase where GAMP is sub-
error floor. In the real-valued input AWGNC case, an error optimal.Moreover,theuseofHadamard-basedoperatorshave
floor always exists but it can be made arbitrary small by a performance cost w.r.t Gaussian ones but which vanishes as
increasingB [2,12]:lim E =lim SER =0,SER B increases; they both have the same algorithmic threshold
B B
→∞ ∗ →∞ ∗ ∗
the error floor in the SER sense. In contrast, in the BEC, ZC forB largeenough(butstillpractical,B 64beingenough).
≥
and BSC cases (more generally for binary input memoryless ConsiderGaussianmatrices.Aninterestingfeatureisthatin
channels), we now prove that as L the error floor constrastwiththeAWGNCcase[12],R forthesebinary
GAMP
→ ∞
vanishes for any (cid:15) and B. This implies that when E =E(cid:101) input channels is not monotonously decreasing; it increases
∗
optimal decoding is asymptotically perfect, and thus GAMP until some B (that may be large) but, although it may be
decoding as well for R R . This is actually verified hard to observe numerically (except for the BEC), it then
GAMP
≤
in practice for GAMP where perfect decoding is statistically decreases to reach lim R = (01)/(2ln(2))<C
B GAMP
→∞ F |
possible even for moderate block-lengths, see blue and red [2]. However, a gap to capacity C persists as long as spatial
curves of Fig. 1. coupling is not employed.
Spatial coupling allows important improvements towards
Spatial coupling allows important improvements towards ShannonCapacity
CRcRcCooppuuo[o[pp2t2tll]]eeee..ddvvTTeeSSnnhhSSeeiinnccmmooppdidirrsseeaammsscctatauuiittccnnccaadhdhlleessrbrbeeeeGGttttttwwiiAAnneeMgMgeessnnP,P,ccRRddooeennppccfiofiooottrrddmmiaiannininnnggddggaaRRststhhGcGclleeiiAmAmuuMMnBnBPiPi→!vvee∞1iirrssssaaRRddlliiuupptteyeoyottoot=t=ooff 2.53 RRRRpGGGcoAAAtMMMPPP,,,LLL===2221991,,,RHHaaandddaamommaarrMddMaMtraaitctrerisciceess
fifinniittee ssiizzee eeffffeeccttss wwhhiicchh aarree mmoorree eevviiddeenntt iinn ccoouupplleedd ccooddeess R
(bothchainlengthsandcouplingwindowsshouldgotoinfinity
(bothchainlengthsandcouplingwindowsshouldgotoinfinity
aafftteerrLLffoorrRRGcGcAAMMPP ttoossaattuurraattee RRppoott)).. 2
AACCKKNNOOWWLLEEDDGGMMEENNTTSS 1.5
We acknowledge Nicolas Macris, Florent Krzakala and
RüWdiegearcUknrboawnlkeedgfoerNheilcpoflualscoMmamcreisn,tsFalsorweneltl aKsrzAalkpaelraKaonsde 0.9 22 24 26 B 28 210 212 214
Rüdiger Urbanke for helpful comments as well as Alper Kose
and Berke Aral Sonmez for an early stage study of GAMP.
and Berke Aral Sonmez for an early stage study of GAMP. 0.8
ThisworkwasfundedbytheSNSFgrantno.200021-156672.
ThisworkwasfundedbytheSNSFgrantno.200021-156672.
0.7
RREEFFEERREENNCCEESS R
[1] A.BarronandA.Joseph,“Towardfastreliablecommunicationatrates 0.6
[1] A.BarronandA.Joseph,“Towardfastreliablecommunicationatrates
nneeaarrccaappaacciittyywwiitthhggaauussssiiaannnnooiissee,,””iinnIInnffoorrmmaattiioonnTThheeoorryyPPrroocceeeeddiinnggss ���ssseeeeeeddd 111 pppJJJ 000
((IISSIITT)),,22001100IIEEEEEEIInntteerrnnaattiioonnaallSSyymmppoossiiuummoonn,,JJuunnee22001100,,pppp..331155––331199.. 0.5
[2] J. Barbier, M. Dia, and N. Macris, “Threshold Saturation of Spatially
[2] J. Barbier, M. Dia, and N. Macris, “Threshold Saturation of Spatially
CoupledSparseSuperpositionCodesforAllMemorylessChannels,”in
CInofuoprmleadtiSopnaTrsheeoSruypewrpoorkssithioonpC(IoTdWe)s,f2o0r1A6llIEMEeEm,o2r0y1l6es.sChannels,”in 0.4 LLLrrr
[3] IAn.foJromsaetpihonaTnhdeoAr.yRw.orBkashrroopn,(I“TFWas),t2s0p1a6rseIEsEuEp,er2p0o1s6it.ion codes have 0.75 22 24 26 B 28 wwwbbb210 www2ff1f2 214
[3] A. Joseph and A. R. Barron, “Fast sparse superposition codes have
nearexponentialerrorprobabilityforR<C,”IEEETans.onInformation
nearexponentialerrorprobabilityforR<C,”IEEETans.onInformation 0.7
Theory,vol.60,no.2,pp.919–942,2014.
Theory,vol.60,no.2,pp.919–942,2014.
[[44]] AiAte..rRaRt.i.vBBelaayrrrrooopnntimaannaddleSSs..tiCmChhaoote,,s“,“”HHiinigghIhn--rfroaatrteemasspptiaaorrnsseeThssueuoppreeyrrppPoorssoiittciieooenndiccnoogddsee(ssISwwITiitt)hh, 0.65 LLLccc
iterativelyoptimalestimates,”inInformationTheoryProceedings(ISIT),
22001122IIEEEEEEIInntteerrnnaattiioonnaallSSyymmppoossiiuummoonn.. IIEEEEEE,,22001122,,pppp..112200––112244.. R0.6
[5] D. L. Donoho, A. Maleki, and A. Montanari, “Message-passing algo-
[5] D. L. Donoho, A. Maleki, and A. Montanari, “Message-passing algo-
rithms for compressedsensing,” Proceedings of theNational Academy 0.55
rithms for compressed sensing,” Proceedings of the National Academy
ofSciences,vol.106,no.45,pp.18914–18919,2009.
[6] oFf.SKcireznackeasl,a,voMl..10M6,éznaor.d4,5,F.ppS.a1u8ss9e1t4,–Y18.9S1u9n,,20a0n9d. L. Zdeborová, 0.5
[6] F. Krzakala, M. Mézard, F. Sausset, Y. Sun, and L. Zdeborová,
“Probabilisticreconstructionincompressedsensing:Algorithms,phase
“Probabilisticreconstructionincompressedsensing:Algorithms,phase 0.45
diagrams, and threshold achieving matrices,” Journal of Statistical
diagrams, and threshold achieving matrices,” Journal of Statistical
[7] MMJ.eeBcchahraabnniieiccrss:a:nTTdhheeFoo.rrKyyraaznankddaElEaxx,pp“eeRrriiemmpeleinnctta,,avvnooall..lyPPs00is8800a00n99d,,a22p00p11r22o..ximatemessage 22 24 26 B0 .2388 210 212 214
[7] Jp.aBssainrbgiedreacnoddeFr.fKorrzasukpalear,p“oRsietipolnicacoadneasl,y”siisnaInndfoarpmpartoixoinmTahteeomryessPargoe- 0.5
pcaesesdiinnggsd(eIcSoIdTe)r,2fo0r14suIpEeErpEoIsnittieornnactoiodneas,l”SiynmIpnofsoirummatoionn,2T0h1e4o.ry Pro- R0.34
ceedings(ISIT),2014IEEEInternationalSymposiumon,2014. RGAMP,SE
[[88]] CsCu..pRReurupssohhs,,iAtAio..nGGcrreoeiidgge,,saavnniddaRRa.p.pVVreeonnxkkiamattaaarrtaeammmaaennsaasnna,,g““eCCpaaappsaascciniittgyy--daaeccchhoiieedvviniinnggg,”ssappraaXrrssieev 0.45 0.3 RRGGAAMMPP,,LL==2299,,RHaandd..MMatartirciecses
[9] sppMurrepe.ppeBrrrpiiannoyttsaiaattiriroXXaniinvvdc::1o1A5d50e0.s11M..v00i5o5a8n89at9ap22np,,ar22roi00x,11i“m55T..ahteemdyensasamgiecspaosfsimngesdseacgoedipnags,s”inagrXoinv R0.4 22 24 B 26
[9] M. Bayati and A. Montanari, “The dynamics of message passing on
densegraphs,withapplicationstocompressedsensing,”IEEETransac-
densegraphs,withapplicationstocompressedsensing,”IEEETransac-
ttiioonnssoonnIInnffoorrmmaattiioonnTThheeoorryy,,vvooll..5577,,nnoo..22,,pppp..776644––778855,,22001111.. 0.35
[10] A. Javanmard and A. Montanari, “State evolution for general approxi-
[10] A. Javanmard and A. Montanari, “State evolution for general approxi-
matemessagepassingalgorithms,withapplicationstospatialcoupling,”
mInafotermmaetsisoangeanpdasIsnifnegreanlgceo,ri2th0m13s.,withapplicationstospatialcoupling,” 0.3
[11] IJn.fBoramrbaiteiro,nCa.nSdchIünlfkeree,nacned,2F0.1K3r.zakala,“Approximatemessage-passing 22 24 26 B 28 210 212 214
[11] J.Barbier,C.Schülke,andF.Krzakala,“Approximatemessage-passing
with spatially coupled structured operators, with applications to com-
with spatially coupled structured operators, with applications to com- FFiigg.. 33:: PPhhaassee ddiiaaggrraammss finorth(fero(mR,tBop))pthlaenAfWorG(fNroCmwtoitphtsonrb=ott1o0m0),
pressed sensing and sparse superposition codes,” Journal of Statistical
[[1122]] JpMaMaaJ.n.rnreeedXdBscBcishhvaaeccaa:rrdaa1bnbnpp5iiiisecaeca0erscrsc3n:i:i.tstaa0yTyTinn8n--hhdada0geecc4oohhaFF0rrini..,yeyedvv2KKaaii0snnnnrr1pgzdgzd5aaar.EkEksssaaxepxpllppaaaaserer,,ussrrepeii““mmeAAresespppununpopptptr,sr,eeooivrvtrxxippoooiioolmlmn..ssaiai22ctttti0i0oeeoo11dnn55emm,,sce,cen”nososoossddJ..aaeoegg55ssuee,,,,”r”--n22ppa00aaaal1s1srrss55XXoii..nnfiivvggStpapddrtreeieeccspptooirrddciinaenerrtlt tifrt✏pLhshouoe=ernotmenbBAni0atnEtaWl.igl1iaCy,nG.lGe,idsbdTNZAeyihCfiCfnMerensoqtawePmLeundaiadotd!hfttvhioBneoersrgbSn1LptC1raiot0→is=tna0tereltnla∞w1ditnnw0ioashs,0iitlettiam,irhsbnoeaiycinn(cid:15)nnfeid=oessmctqreut0fauhfiaor..aved1nrtRe.iieontBeGTabgREcAhtLhapCieMtio=s,n(tLPeRZt2,wdi→C9s,foBofboo∞ramy)mrbntidfiaraantunlrniilnaBndmiyntenSedasbidCni.yLetgifRfior=daonGnGlelme2fiAAdRw9nMMiptfibnhoPtoPhygetr,
[[1133]] JaJ..rXBBiaavrr:bb1ii5ee0rr,3,.MM08..0DD40iiaa,,,2aa0nn1dd5.NN..MMaaccrriiss,,““PPrrooooffooffTThhrreesshhoollddSSaattuurraattiioonnffoorr tohve!ertr1a1n0s0itiionnstaasnctehsehfoigrheeascthra(teRf,oBr)whanicdhbaytldeaesfitn5in0ginthsteantcraensswitieorne
SSppaattiiaallllyyCCoouupplleeddSSppaarrsseeSSuuppeerrppoossiittiioonnCCooddeess,,””iinnIInnffoorrmmaattiioonnTThheeoorryy dasectohdeehdig(huepsttorataesfmoralwlhSicEhRatduleeastot 5fi0niintestasinzceesefwfeecrtes)s.uTccheessifnunlleyr
PPrroocceeeeddiinnggss((IISSIITT)),,22001166IIEEEEEEIInntteerrnnaattiioonnaallSSyymmppoossiiuummoonn,,22001166.. fidegcuoredeidllu(sutpratteosafinsimtealslizSeEeRffedcutsebtoy cfionmitepasriiznegeRffGeActMs)P. Tcohmepinunteedr
[14] S. Rangan, “Generalized approximate message passing for estimation
[14] Sw.itRharnagnadno,m“Glineneaerramlizixedinga,p”parorXxiimvaptreepmrienstsaagrXeivp:a1s0si1n0g.5f1o4r1,es2t0im12a.tion ifinguthriesiwlluasytr(aftoerstfihneitBeSsCiz)eaenfdfetchtes“btyruceo”mLp→arin∞g RcuGrvAeMpPrecdoimctepdutbedy
withrandomlinearmixing,”arXivpreprintarXiv:1010.5141,2012. SinEt;htihsewfianyit(efoLr tthreanBsiStiCo)nafnodllothwes“vtreurye”cLlosely thceuravseymprpetdoitcitcedonbey.
[[1155]] CaCp..pCCroooanncddhoo,”aainnnddSiWWgn..aJJl..PGGrorroocssesss,,si““nSSgppSaaryrssseetemssuusppee(SrrppiPoosSsii)tt,iioo2nn01cc5ooddIeeEssE::EAAWpporrraakccstthiiccoaapll TSEhe; ttwheofiRnGitAeMLPtcraunrvsietsio(ndafsohlleodwasndvesroylicdl)oi!slelul1ystrtahteeathsyatm,pdteostpiciteotnhee.
aoopnnp,,rOOoaccctth22,”0011i5n5,,Sppippg..n1a1–l–66P..rocessing Systems (SiPS), 2015 IEEE Workshop mThisemtwatochRiGnAthMePractuersvebset(wdaesehnetdheanHdasdoalmida)ridl-lubsatsreadtectohdaitndgesmpaittericfoesr
[16] J.Barbier,M.Dia,N.Macris,andF.Krzakala,“TheMutualInformation alonwdtBheHGaaduasmsiaanrdo-bnaessefdorcolodwinBg,mbaottrhicreasteasllcoowinctoiderefaocrhlalrogweeBr.raTthees
[16] J.Barbier,M.Dia,N.Macris,andF.Krzakala,“TheMutualInformation
inRandomLinearEstimation,”ininthe54thAnnualAllertonConfer- rtheganionGabuestswiaenenonthees,rtehdeyanqduicbklulyebceucrovmeseiisndthisetinhgarudishpahbalsee.frToomfitnhde
inRandomLinearEstimation,”ininthe54thAnnualAllertonConfer-
eenncceeoonnCCoommmmuunniiccaattiioonn,,CCoonnttrrooll,,aannddCCoommppuuttiinngg,,SSeepptteemmbbeerr22001166.. RGaGcuAsMsiPan, wmeatrfiocleloswpetrhfeormsaamneceps,robcuetduarlleowastoforreRacGhAlMarPgebruBt .usTinhge
[[1177]] GG.. RReeeevveess aanndd HH.. DD.. PPfifisstteerr,, ““TThhee rreepplliiccaa--ssyymmmmeettrriicc pprreeddiiccttiioonn ffoorr srepgaitoianllybectwouepenledtheHareddamaanrdd-bblauseedcuorpveersatdoerlsimaintadteLth=e h2a1r1d. pThhaessee.
IccInnoottmmeerrppnnrraaeetstsiissooeennddaallsseSeSnynysmsmiinpnpgogossiwiwuuimimtthhoogngnaaIIuunnssffssooiirarammnnaammttiiooaantntrriiTTccehehsseeooiirrssyyee((xxIISSaaIcIcTTtt,,))””,,JJiinunullyy2200221100661166II..EEEEEE apFbrauoertracumfiosnenindtsegitnrrsugs,cptsRaeetdGceiaAalmlsMyiddPcde,olseuwcprieilnbenfedoedlrHliofinawgd[ua1trmh1ee,af1rosd2ar-]mb,thaweeseiptbdhrloootcchpekeedrdufaoerteolcloroasmwsapifnnoodgsricLtRioo=GunpA2olMi1fn1Pga.
[18] S. Kudekar, T. Richardson, and R. L. Urbanke, “Spatially coupled
[18] SeeTnn.rsaseKenmmsu.bdbolelenekssaInruu,fnnoiiTrvvm.eerraRsstaaiiloclllhnyyaTaardchchsheoioieenrvvy,ee,avccnoaadlpp.aa5Rcc9ii.tt,yynLuou.n.nd1dUe2errr,bpbabpnee.lklii7eee7,ff6pp“1rrS–oop7ppa8aatg1giaa3altt,liiyoDonnec,,”c”ou2IIEEp0lE1Ee3EEd. ncTcoouhuumeppsbleleiendragrocecofodcbnoilnsongtcsrtkuorc-uprtceoiotrweandstoaLarn:srdnd=ufeomsLlclbcoreiw+brie1ondf;gbbinclaooc[cuk1kpw1-lci,ano1rgdl2u]pam,nawnrdasimtfLhoerctawe∈rads{ri:d8sy#,cm1o6bmu,lp3oelt2cirnk}igc-;
Trans.onInformationTheory,vol.59,no.12,pp.7761–7813,Dec2013. wcoilnudmownssLwcb2∈{8{,21,63,,352,}7;}#, wbflo∈ck-{r1o,w2s};Lrco=upLlcin+g1s;trbeancgktwha√rdJ(i.∈e
[b0e.f5o3r,e0t.h7e3d]ifaogrotnhaelAbWlocGkNs,Cw,i0th.3oufotrcothuenotitnhgerthcehadniangeolsna(allblltohcekbsl)oacnkds
ofothrwerarthda(ninthferolnigt)htcboluupelicnoguwpliinndgoownseswandth2e,3a,ll5-,z7ero,swblocks1h,a2ve;
b f
uconuitpslitnregngstthre)n;grheltatoivfethseizefoorwfathrde “bsleoecdk”s2bpl{oJck β[s0e.e5d3},0[1.7.0322],f{1o.r25th]}.e
2 ∈
AWGNC, 0.3 for the other channels (all the blocks other than the
forwardcouplingoneshaveunitstrenght);relativesizeofthe“seed”
block � [1.02,1.25].
seed
2