MULTI-PROCESSOR APPROXIMATEMESSAGEPASSING USINGLOSSY COMPRESSION PuxiaoHan,v JunanZhu,n RuixinNiu,v andDrorBaronn v:VirginiaCommonwealthUniversity,Dept.ElectricalandComputerEngineering Richmond,VA23284,U.S.A.Email:{hanp,rniu}@vcu.edu n:NorthCarolinaStateUniversity,Dept.ElectricalandComputerEngineering 6 Raleigh,NC27695,U.S.A.Email:{jzhu9,barondror}@ncsu.edu 1 0 2 ABSTRACT Comparedwithmanyresultsondistributedcomputation, n a Inthispaper,acommunication-efficientmulti-processorcom- optimization,andnetworktopology[3,13]inMP-CS,amod- J pressed sensing framework based on the approximate mes- estsubsetoftheliteratureconsidersthecommunicationcosts 8 sagepassingalgorithmis proposed. We performlossycom- of the GC step [4,5,14,15]. In this paper, we still consider 1 pressiononthedatabeingcommunicatedbetweenprocessors, multi-processor AMP (MP-AMP), and focus more on the ] resultinginareductionincommunicationcostswithaminor communication costs. In contrast to our prior work [6], we C degradationinrecoveryquality. Intheproposedframework, are willing to accept a minor decrease in recovery quality D a new state evolution formulation takes the quantization er- while providing significant and often dramatic communi- . rorintoaccount,andanalyticallydeterminesthecodingrate cations savings. Such results are especially well-suited to cs requiredineachiteration. Twoapproachesforallocatingthe clusterswherecommunicationbetween serversis costly. To [ codingrate,anonlineback-trackingheuristicandanoptimal achieve the reduction in communicationcosts, we use lossy allocation scheme based on dynamic programming, provide compression to reduce the inter-processor communication 1 v significantreductionsincommunicationcosts. costs, and provide a modified SE formulation that accounts 5 for quantization error. Two approaches for allocating the Index Terms— lossy compression, multi-processor ap- 9 codingrate,anonlineback-trackingheuristicandanoptimal 5 proximatemessagepassing,ratedistortionfunction. allocation scheme based on dynamic programming, provide 4 significantreductionsincommunicationcosts. Furthermore, 0 1. INTRODUCTION we consider Bayesian AMP, which achieves better recovery . 1 accuracy than non-Bayesian AMP [7] by assuming that the 0 Compressedsensing(CS)[1,2]hasnumerousapplicationsin unknownsignalfollowsaknownpriordistribution. 6 variousareasofsignalprocessing.Duetothecurseofdimen- Inthe following,boldcapitalandboldlower-caseletters 1 : sionality,itcanbedemandingtoperformCSonasinglepro- are used to denote matrices and vectorsrespectively, capital v cessor. Furthermore,clusterscomprisedof manyprocessors letters without bold typically refer to dimensionality or ran- i X have the potential to accelerate computation. Hence, multi- dom variables, and []T denotes vector or matrix transposi- · r processorCS(MP-CS)hasbecomeofrecentinterest[3–5]. tion. The ℓ2 norm of a vectoris denotedby , (µ,σ2) a We considerMP-CSsystemscomprisedoftwoparts: (i) isa Gaussiandistributionwithmeanµ andvkar·iakncNeσ2, and local computation (LC) is performed at each processor, and [a,b]isacontinuousuniformdistributionwithin[a,b]. U (ii) global computation (GC) obtains an estimate of the un- known signal after processors exchange the results of LC. 2. THECENTRALIZEDAMPALGORITHM In our previous work [6], we developed an MP-CS framework based on the approximatemessage passing (AMP) al- Approximatemessagepassing(AMP)[7]isastatisticalalgo- gorithm [7], in which a GC approach performs AMP in an rithmderivedfromthetheoryofprobabilisticgraphicalmod- MPsystem,providingthesamerecoveryresultascentralized els [16]. Given noisy measurements y = As + e of the 0 AMP.WechoseAMP,becauseitisanalyticallytractabledue unknownsignals RN,whereelementsins areindepen- 0 0 to the state evolution (SE) [8,9] formalism, and can be ex- ∈ dentandidenticallydistributed(i.i.d.) realizationsofascalar tended to Bayesian CS [9,10], matrix completion [11], and randomvariable S p , A RM×N is the sensing ma- non-negativeprincipalcomponentanalysis(PCA)[12]. trixwithentries 0i.i∼.d. S0(0,1/∈M),ande RM isadditive ∼ N ∈ measurementnoise,whichisi.i.d. (0,σ2),AMPiteratively TheworkofP.HanandR.NiuwassupportedinpartbytheVCUPres- N e identialResearchQuestFund. TheworkofJ.ZhuandD.Baronwassup- recoverss0,startingfromaninitialestimatex0 =0andresid- portedinpartbytheNationalScienceFoundationunderGrantCCF-1217749 ualz =y: andtheU.S.ArmyResearchOfficeunderContractW911NF-14-1-0314. 0 ft =xt+ATzt, (1) xt+1 =ηt(ft), (2) ItcanbeseenthatintheGCstepofMP-AMP,eachpro- zt+1 =y−Axt+1+(N/M)ηt′(ft)zt, (3) csuesmsosrthpemsentodsobfttpaitnofthaenfduxsion,ceanntders,enanddsxthe futosioeancchepnrtoer- t t+1 t+1 wheretistheiterationnumber,thebarabovethevectorin(3) cessor.1 Ourgoalinthispaperistoreducethesecommunica- denotes its empirical average, η is known as the denoising t tioncostswhilebarelyimpactingrecoveryperformance. functionordenoiser,andηt′ denotesitsderivative. Supposethatalltheelementsinfparecomputedas32-bit t Accordingto BayatiandMontanari[9], asN and single-precisionfloating-pointnumbers.BecauseSEisrobust → ∞ M/N = κ > 0, the elementsof ft in (1) followi.i.d. Ft = to small perturbations[7,8], we can compress fp lossily up S +σ Z,whereZ (0,1)andthesequence σ2 satisfies t 0 t ∼N { t} tosomereasonabledistortionlevel,andsendthecompressed σt2+1 =σe2+(1/κ)E[ηt(S0+σtZ)−S0]2 (4) outputtothefusioncenter.Toensurethatthiserrorisindeeda =σ2+(1/κ)E x s 2/N. “smallperturbation,”werequiretheerrortobeadditiveand, e k t+1− 0k if possible, white and Gaussian, so that we can analyze the Notethatσ02 = σe2+(1/κ)E[S0]2;equation(4)isknownas relationshipbetweentheerrorandcodingrate. stateevolution(SE),andtheoptimaldenoiserformeansquare error(MSE)istheconditionalmean[9,17]: 3.2. LossyCompressionoffp t ηt(Ft)=E[S0 S0+σtZ =Ft]. (5) Before we propose specific lossy compression approaches, | we describe an important property of MP-AMP. In addition In this paper, we assume that S follows the Bernoulli 0 to the well-knownGaussianity of the vectorf s [9], nu- Gaussiandistribution: mericalresultsshowthatelementsoffp (1/tP−)s0are also pS0(s)=ǫN(s;µs,σs2)+(1−ǫ)δ(s), (6) fi.pi.d.G(1a/uPss)isanwanidthfmqean(10/aPnd)svarairaencinetdσ−et2p/ePnd.eFnutrf0tohrerdmifoferer-, whereδ(s)denotestheDiracdeltafunction,andS0typically etnt−processor0s p andt q−. In light0of this property, fp can be t hasmeanµs =0. Thedenoiseriseasilyderivedusing(5). describedasascalarchannel: Asameasureofthemeasurementnoiselevelandrecovery Fp =S /P +(σ /√P)Z , whereZ (0,1). accuracy,wedefinethesignal-to-noise-ratio(SNR)as t 0 t p p ∼N FortheBernoulliGaussiandistribution(6), SNR=10log10 E kAs0k2 /E kek2 Ftp ∼ǫN µs/P,(σs2+Pσt2)/P2 +(1−ǫ)N 0,σt2/P . 10log E (cid:0)s (cid:2)2 /E e(cid:3) 2 (cid:2) =10(cid:3)(cid:1)log ρ/σ2 , Scalar Qu(cid:0)antization: Next, we (cid:1)propose a uni(cid:0)form qua(cid:1)n- ≈ 10 k 0k k k 10 e (cid:0) (cid:2) (cid:3) (cid:2) (cid:3)(cid:1) (cid:0) (cid:1) tizerwithentropycoding,alsoknownasentropycodedscalar where ρ = ǫ/κ, and the signal-to-distortion-ratio (SDR) at quantization(ECSQ)[18]. iterationtas LetΨ(u)denotethecharacteristicfunctionofFp, itcan t UsingStDheRS(tE)e=qu1a0tiloong1in0(cid:0)(4E),(cid:2)kwse0kh2a(cid:3)v/eE(cid:2)kxt−s0k2(cid:3)(cid:1). bes|hΨo(wun)|th≤atǫexp −0.5 σs2+Pσt2 u2/P2 SDR(t)=10log10 ρ/ σt2−σe2 . +(1−ǫ)exp(cid:0)(cid:2)−0.5σ(cid:0)t2u2/P(cid:1)≤(cid:1)exp(cid:0)−0(cid:3).5σt2u2/P(cid:1) NotethattheBernoulliGaussian(cid:2) as(cid:0)sumption(cid:1)i(cid:3)n thispaperis isnearlyband-limited. Dueto thisproperty,itispossible to onlyforillustration,andourworkiseasilyextendedtoother develop a uniform quantizer of fp i.i.d. Fp, where the priordistributionsp . quantizationerrorvpisapproximattel∼ystatisticaltlyequivalent S0 t to a uniformly distributed noise Vp [ 0.5∆ ,0.5∆ ] uncorrelatedto Fp. Actually, a qutan∼tizaUtio−nbin sQize ∆ Q 3. MULTI-PROCESSORAMPFRAMEWORK t Q ≤ 2σ /√P willsufficeforvalidationofvp i.i.d. Vp [19]. t t ∼ t 3.1. CommunicationinMulti-ProcessorAMP The fusion center will receive the quantized data fp t ∼ i.i.d.Fp,andcalculatef = P fp i.i.d. F ,where Consider a system with P processors and one fusion cen- t t p=1 t ∼ t e P P P tnearm. EelaycAhppr,oacnedssoobrtapin∈sy{p1,=··A· ,pPs}+taekpes. TMhe/PprorocwedsuorefsAin, e Ft = Ftp =eFt+Vt, aendVt = eVtp. (7) 0 Xp=1 Xp=1 (1)—(3)canthenberewritteninadistributedmanner: e e Applyingthecentrallimittheorem,V approximatelyfollows LocalComputation(LC)performedbyeachprocessorp: t (0,Pσ2)forlargeP,whereσ2 =∆2/12. zpt =yp−Apxt+(1/κ)ηt′(ft−1)zpt−1, ENntropyQCoding and OptimumQ Bit RQate: Let pi be the probabilitythatFpfallsintothei-thquantizationbin.Theen- fp =x /P +(Ap)Tzp. t t t t tropyofquantizedFp, Fp, isH = p log (p )[20], t t Q − i i 2 i GlobalComputation(GC)performedbythefusioncenter: P 1In order to calculate eachezp , the fusion center also needs to send P t+1 ft = ftp, ηt′(ft), andxt+1 =ηt(ft). ηnit′c(aftti)ontocaolslttihsenpegrolicgeisbsloercs.omThpiasreisdawsicthaltahra,taonfdttrhaenscmoritrteisnpgoandviencgtocro.mmu- Xp=1 thatis, the sensors need H bits on averageto presenteach toalltheprocessors. Thecorrespondingcommunicationcost Q elementinftptothefusioncenter,whichisachievablethrough isalsonegligiblecomparedwiththatofcommunicatingftp. entropycoding[20]. e Inratedistortion(RD)theory[20],wearegivenalength- 3.4. DynamicProgramming(DP-MP-AMP) nrandomsequenceY = Y n i.i.d. Y,andourgoal n { n,i}i=1 ∼ While back-trackingis a usefulheuristic, it is possible for a istoidentifyareconstructionsequenceY = Y n that n { n,i}i=1 givencodingbudgetRperelement,totalnumberofAMPit- canbeencodedatlowrate whilethedisbtortionbd(Yn,Yn) = erationsT,andinitialnoiselevelσ2 inthescalarchannelto n1 id(Yn,i,Yn,i) (e.g., squared error distortion) bebtween compute the coding rate allocation0s among the AMP itera- thePinput and reconstruction sequences is small. RD theory tionsthatminimizethefinalMSE,σ2 . b T,D hascharacterizedthefundamentalbest-possibletrade-offbe- Todoso,notethatwecanevaluateσ2 offlineandhence tween the distortion D = d(Y ,Y ) and coding rate R(D), t,C n n obtain the number of iterations required to reach the steady which is called the rate distortion function. The RD func- b state, which would be a reasonable choice for T. Second, tionR(D)canbecomputednumerically(cf.Blahut[21]and recallingthenewSEequationin(8),σ2 dependsonσ2 Arimoto[22]). Fortheuniformquantizerthatyieldsaquanti- t,D t−1,D andσ2,whichisafunctionofR ,thecodingrateallocatedin zationMSEofσ2 withacodingrateH bitsperelement,the Q t Q Q thet-thiteration.Therefore,wecanrewriteσ2 asfollows: RDfunctionwillgiveabitrateR(D =σ2)80.?@A B7+C.=DE<8D89*;*=F9.FG.16(cid:239)@A(cid:239)?@A & FE*=D;<.:8HI<*H.FG.,A(cid:239)@A(cid:239)?@A5.-3#6 & & B7+C.=DE<8D89*;*=F9.FG.,A(cid:239)@A(cid:239)?@A !!. " # $ % & ' ( ) !! " # $ % & ' ( ) L "! !! # % ' ) "! "# "% "' ") #! * * * ' . ' ' & & & D89*2% % % J=*.:;*8./J=*H.E8:.8<8#$ -BFBE77,*++=.DECC:;8..<==0.DD:=8KEEH*=<

Multi-Processor Approximate Message Passing Using Lossy Compression

Similar Multi-Processor Approximate Message Passing Using Lossy Compression