Advanced Mean Field Methods: Theory and Practice PDF
Preview Advanced Mean Field Methods: Theory and Practice
AdvancedM ean FieldM ethods NeuralI nformatioPnr ocessinSge ries MichaeIl.J ordaSna,r aI .S olla AdvanceisnL argMea rgiCnl assifiers AlexandJe.Sr m olaP,e te1r. B artleBtetr,n haSrcdh olkkoapnfdD, a leS chuurmans, eds,2. 000 AdvanceMde anF ielMde thodTsh: eorayn dP ractice Manfred OpapnedrD avidS aade,d s,2. 001 AdvancedM ean FieldM ethods Theorayn dP ractice Editebdy ManfreOdp pearn dD aviSda ad TheM IT Press CambridMgaes,s achusetts LondonE,n gland © 2001M assachuseItntsst ituotfTe e chnology Allr ightrse serveNdo. p arto ft hisb ookm ay be reproducienda ny formb y any electronic orm echanicmaela ns (inclupdhiontgo copyirnegc,o rdinogri, n formatisotno ragaen d retrieval) withoupte rmissiionnw ritinfgr omt hep ublisher. LibraroyfC ongresCsa taloging-in-PublDiactaat ion Advancedm ean fieldm ethods: theorayn d practice/edbiyt ed ManfreOdp pera ndD avidS aad p. cm.-(NeurIanlf ormatiPorno cessiSnegr ies) Includbeisb liograprheifcearle nces. ISBN 0-262-1505(4a-l9kp .a per) I.M ean fieldt heory II.O pperM,a nfredI.I IS.a ad,D avid. QC174.85.MA4338 2001 530.15'95-dc21 00-053322 CONTENTS SerieFso reword vii Foreword viii Contributors xi Acknowledgments xiv 1 Introduction 1 ManfreOdp pera ndD avidS aad 2 From NaiveM ean FieldT heoryt ot heT AP Equations 7 ManfreOdp pera ndO leW inther 3 An IdiosyncraJtoiucr neyB eyondM ean FieldT heory 21 JonathaSn.Y edidia 4 Mean FieldT heoryf orG raphicaMlo dels 37 HilbeJr.tK appena ndW im J.W iegerinck 5 The TAP Approacht o Intensivaen d Extensive ConnectiviStyys tems 51 YoshiyuKkaib ashimaan dD avidS aad 6 TAP ForP arityC heckE rrorC orrectinCgo des 67 DavidS aadY,o shiyuKkaib ashimaan dR enatVoi cente 7 AdaptivTeA P Equations 85 ManfreOdp pera ndO leW inther 8 Mean-fieldT heoryo fL earningFr:o m Dynamicst o Statics 99 K.Y . Michael SW.oL niag n,d P eixuLnu o 9 Saddle-poiMnett hodsf orI ntractabGlrea phical Models 119 FernandJo.P inedaC,h eryRle scahn dI -JenWga ng 10 Tutorialo n VariationAaplp roximatioMne thods 129 Tommi S.J aakkola 11 GraphicaMlo delsa nd VariationMaelt hods 161 ZoubiGnh ahramaannid M attheJw. B eal 12 Some Exampleso f RecursivVea riational Approximatiofnosr B ayesiaInn ference 179 K.H umphreyasn dD .M. Titterington 13 TractableA pproximatBee liePfr opagation 197 DavidB arber vi Contents 14 The AttenuateMda x-ProducAtl gorithm 213 BrendaJn. F reya ndR alKfo etter 15 Comparingt heM ean FieldM ethod andB elief Propagatiofno rA pproximatIen ferencien M RFs 229 YaiWre iss 16 InformatioGne ometryo fa -ProjectionM eainn Field Approximation 241 Shun-icAhmia riS,h irIok edaan dH idetosShhii mokawa 17 InformatioGne ometryo fM ean-FielAdp proximation 259 ToshiyuTkain aka SERIEFSO REWORD ( ) The yearlNye uraIln formatPiroonc essSiynsgt emNsI PS workshobprsi ntgo gethesrc ientwiisrtbhsr oadvlayr yibnagc kgrouinnds st atistmiactsh,e maticcosm, putesrc iencpeh,y sicesl,e ctreincgailn eernienugr,o scieanncdec ,o gnitsicviee nce, unifiebdy a commond esitroed evelnoopv eclo mputatioannadsl t atistsitcraalt e giefso ri nformatpiroonc essainndgt ,o u nderstatnhdem echanisfmosri nformation processiinnt gh eb rainA.s opposetdo c onferentcheess,we o rkshompasi ntaai n flexibfloer matth abto tha llowansd e ncouratgheesp resentaatnidod ni scussoifo n worki np rogreasnsd,t huss ervaes a ni ncubatfoorrt hed evelopmoefni tm portant newi deaisn t hirsa pidelvyo lvifinegl d. TheS eriEedsi torisnc, o nsultawtiitohwn o rkshoopr ganizaenrdsm emberosf theN IPSF oundatiBoona rds,e lescpte ciwfiocr kshotpo picosn theb asiosf s ci entifiecx celleinnctee,l lecbtrueaald tahn,d t echniicmapla cCto.l lectoifop nasp ers choseann de ditebdy theo rganizoefrs sp eciwfiocr kshoaprse b uiladr ounpde d agogical introductowrhyi lcrehe aspetaerrcsh, monopgrroavpihcdsoe m prehensive descriptoifow nosr kshop-retloaptiecdts o,c reatae s erioefsb ooktsh atp rovides a timelayu,t horitaatcicvoeu notft hel atedsetv elopmeinntt sh ee xcitifinegl do f neuraclo mputation. MichaeIl.J ordaSna,r aI .S olla FOREWORD Thel inkbse tweesnt atistpihcyaslia cnsd t hei nformatsicoine nces-inccloumding putesrc iensctea,t istaincdcs o,m municattihoeno ry-hagvreo wsnt rongienrr e cent yearass,t hen eedosf a pplicathiaovneis n creasilnegdrl eys earcihnet rhsei nforma tiosnc iencteosw ardtsh es tudoyf l arge-shcialgeh,l y-coupprloebda bilsiyssttiecm s thaatr er eminiscoefmn otd elisn s tatistpihcyasli cOsn.eu sefluiln iks t hec lasosf MarkovC hainM onteC arl(oM CMC)m ethodssa,m pling-baalsgeodr itwhhmoss e rootlsi ien t hes imulatoifog na seasn dc ondensmeadt tebru,t w hose appealing gen eraliatnyds implicoifit myp lementahtaivoen spanrekwea dp plicattihornosu ghout thei nformatsicoine ncAenso.t hesro urcoefl inkcsu,r renutnldye rgorianpgi dde vel opmenti,st hec lasosfm ean-fiemledt hodtsh aatr et het opiocf t hibso okM.e an fielmde thodasi mt os olvmea nyo ft hes amep robleamssa rea ddressbeydM CMC methodbsu,t d os ou sindgi fferecnotn ceptaunadlm athematitcoaoll Mse.a n-field methodasr ed eterminimsettihco dmsa,k inugs eo ft oolssu cha sT ayloerx pansions andc onverxe laxatitooa npsp roximoartb eo undq uantitoifie nst ereWshti.l teh e analysoifMs C MC methodrse posoenst het heoroyfM arkocvh ainasn ds tochastic matricmeesa,n -fiemledt hodmsa kel inktsoo ptimizatthieoonr ayn dp erburbation theory. Underlyimnugc ho ft heh eighteniendt ereisntt h eslei nkbse tweesnt atistical physiacnsd t hei nformatsicoine nciests h ed evelopme(nitnt hel attfieerl do)f a generfarla mewofrokr a ssociatjioningpt r obabildiitsyt ributwiiotnhgs r aphs, andf or explotihteis ntgr uctuorfte h eg raphi nt hec omputatioofnm arginal probabilaintdie exsp ectatiPornosb.a bilgirsatpihci mcoadle lasr eg raphs-directed oru ndirected-annowtiatthfe udn ctiodnesfi neodn locacll usteorfns o detsh at whent aketno gethdeerfi nfea miloifej so inptr obabidliisttyr ibuotnit ohnesg raph. Noto nlayr et hec lassimcoadle losfs tatistpihcyaslii cnss tanocfeg sr aphimcoadle ls (generailnlvyo lvuinndgi recgtreadp hs),m abnuyat p pliperdo babilistic models with noo bvioucso nnecttioop nh ysiacrse g raphicmaold elass w ell-exampilnecsl ude phylogenterteieicsn g enetidcisa,g nosstyisct eimnsm ediciunnes,u pervliesaerdn ing modelisn m achinlee arnianngd,e rror-conctordoeilsn i nformattihoeno rTyh.e availabiolfti hteyg enerfarla mewohraks m adei tp ossibfloeri deatso fl ow more readibleyt weetnh esfiee lds. Inp hysiocnse o ft hep rinciappapll icatoifom nesa n-fiemledt hodisst hep redic tioonf " phaster ansiti,do inssc"o ntiniunia tgigerse gaptreo pertoifae ssy steumn der thes calionfog n eo rm orep arametearsss ociawtietdth h es ysteAm .p hysicriesatd ingt hec urrebnoto km ayt husb es urpribsyet dh er elatiivneflrye quoecnctu rrence oft het erm" phaster ansitiIonnt "h.ea pplicattioot nhsei nformatsicoine nciets , iso ftetnh ev alueosft he" microscovpairci"a btlheasta reo fm osti nterewshti,l e the" macroscoppriocp" ertoifet sh es ysteamr eo fteno fs econdairnyt ereTshtu.s int heg enetiacpsp licatwieoa nr ei nteresitnte hde g enotyopfes peciifincd ividuals; int hed iagnosatpipcl icatoiuorni sn teriessi tnt hep robabiloifst pye cidfiics eases; andi ne rror-cocnotdrionlwg e w isht or ecovtehreb itisn t het ransmitmteesds age. Moreoveirnm, a nyo ft hesaep plicatwieoa nrsei nteresitnae sdp ecigfirca phw,h ose Foreword ix parametearrsed eterminbeyds tatistmiectahlo dbsy,a domaienx perotrb ya de signearn,d i ti sa matteorf s econdairnyt ereasstt o h owa ggregaptreo pertoife s thep robabildiitsyt ribuwtoiuolncd h angien s omeh ypothetailctaelr nagtriavpeh inw hicche rtapianr ametehrasv eb eens caled. Thisi sn ott os ayt haatg gregaptreo pertoifpe rso babidliisttyr ibuatrieon nost ofi ntereisntd;e etdh eya rek eyt ou nderstandtihnemg e an-fiealpdp roacThh.e calculaotfit ohnep robabidliisttyr ibuotfai noyng ive"nm icroscovpairci"a ble-the marginparlo babiolfia t nyo dei nt heg raph-iasna ggregatoipoenr ation, requiring summinogr i ntegratthienj go inptr obabilwiittyhr espetcota lolt hevra riablIens . statisttiecramlos n ei sc alculatai "nlgo lgi kelihotohdep" h;y sitcesr minoliosg y the" freeen erg.yI "n t hec omputatiofnraalm eworrekf errteoad b ovoen ea ttempts toe xplotihte c onstraiinmtpso sebdy theg raphicsatlr uctutroce o mputteh ese quantiteiffiecsi entelsys,e ntiuaslilnytg h em issiendgg eisn t heg raptho m anage thep roliferoafti inotne rmeditaetrem tsh ata risien c omputimnugl tipsluem so r integrTahlissa. p proahcahs b eens uccessifnmu aln ya ppliperdo blempsr,i ncipally involvgirnagp hisn t hef ormo ft reeosrc hainFso.r m oreg energarla phhso,w ever, a combinatoerxipallo soifotne rni seusp t os laayn ya ttempttoc alculmaatreg inal probabileixtaicetsl y. Unfortunatietil spy r,e cistehleys ger aphtsh aatr en oti nt hef ormo ft reeasn d chaintsh aatr eo nt her esearfcrho ntiinem ra nya pplifieedl dNse.w i deaasr en eeded toc opew itht hesger aphasn,d r ecenetm pirirceaslu lhtasv es uggestmeeda n-field andr elatmeedt hodass c andidates. Mean-fiemledt hodtsa kea moren umericaaplp roatcohc alculatiinog nrsa ph icamlo delsT.h erea res everwaaly st ou nderstamneda n-fiemledt hodsa,n dt he currebnoto kp rovideexsc ellceonvte raogfea lolf t hep rincippearls pectiOvnees . majort hemei st hato f" relaxatiaonn i"d,e af amilifarro mm oderno ptimization theorRya.t hetrh anc omputian sgp ecipfirco babildiitsyt ribuotnieor ne,l axtehse constradienfitnsi ntgh ep robabidliisttyr ibuotbitoani,n ainno gp timizaptrioobnl em inw hicthh es olutitoont heo riginparlo bleimst he( uniquoep)t imumR.e laxing constraiinntvso lvienst roducLianggr angmeu ltipliaenrdsa ,l goritchamnsb ed e velopiendw hicthh eo rigin"aplr,i maplr"o bleimss olvevdi a" dualr"e lationships amongt heL agrangivaanr iables. Thiso ptimizapteirosnp ectiisiv mep ortatnout n derstantdhienc go mputational consequenocfea sd optitnhge p hysicfsr ameworIkn. particiunlt ahre,p hysics framewotrhke f reeen ergtya keas mathematifcoarlm i nw hichc onstraianrtes readiilmyp oseadn dr eadi"lrye laxeNdo"t.e a lsot hatt hep hysicfsr amework permietxsp resstihnefg r eeen ergays t hes umo ft wot erms-th"ea veraegnee rgy" andt he" entropCyo"m.p utatiomneatlh odcsa nb ed eveloptehda ta reg earetdo thes pecimfiact hematifcoarlm tsa kebny t hestee rms. Theo ptimizapteirosnp ecttihvamete an-fietlhde orbyr intgost het abliesu seful ina nothewra y.I n partictuhleag rr,a phicmaold elsst udieidnt hei nformation sciencaerseo ftenno tf ulldye terminbeyda priosrc ienttihfieco rbyu,t a rev iewed ass tatistmiocdaell tsh ata ret ob e fit too bserved dFaittat.i an gm odelt o datag eneralilnvyo lvseosm ef ormo fo ptimizatiotnh-eis ni mplessett tionnge maximiztehse l ogl ikelihwoiotdhr espetcott hem odelp arameteArss w.e have x Foreword discusstehdem, e an-fiealpdp roancaht uratlrleya tthse l ogl ikelih(ofordee en ergy) asa parameterfiuznecdt itoonb eo ptimizaendd,i tm ightb ee xpecttehda tt his approacwho uldt herefeoxrtee nrde aditloyl ikelihood-sbtaasteids tmiectahlo ds. Indeetdh,e s implemseta n-fiemledt hodysi elad l owebro undo nt hel ogl ikelihood, ando nec anm aximitzhei lso webro unda sa surrogfaottreh e( generailnltyr actable) maximizatoifot nh el ogl ikelihood. Whilael olf t hesaer gumenmtasy h avea ppeatlot hep hysicpiasrtt,i cultahrel y physicciosntt emplautnienmgp loymeinntt h em odern" informaetcioonno m,yf "o r thei nformatsicoine nttihsetri esr oomf ord oubtA. s urveoyft hem odelsst udibeyd thep hysicirsetvsepa rlo perttiheasdt i verfgreo mt hen eedosf t hei nformatsicoine n tisStt.a tistpihcyasli cmaold elasr eo ftehno mogeneous-ptahrea metleirnsk itnhge nodeasr et hes amee verywheirnt eh eg raphM.o reg eneraltlhyep, h ysicmaold els choospea rametefrrso md istribut(i"sopnisn -glmaosdse lsb"u)t t hesdei stributions aret hes amee verywheirnte h eg raphT.h em odelasl lo"wfi eltde rmst"h ata re equivalteon" to bservdeadt ai"n t hes tatistsiectatli bnugt,o ftetnh esfiee ltde rms area ssumeedq ualVa.r iougsr aphiscyamlm etriaerseo fteinn vokeSdo.m em odels assumien finite-racnognende ctiAolnlso .ft hesaes sumptisoenesm r athefra rf rom theh ighliyn homogeneiorurse,g usleatrt ionfgm odelisn s ettinsgusc ha sg enetics, medicdaila gnosuinss,u pervilseeadr nionreg r ror-cocnotdrionlg . Whilei ti sp ossibtlhea ts omeo ft hesaes sumptiaornesr equirfeodrm ean fielmde thodtso s ucceetdh,e raer er easotnosb elietvhea tth es copoef m ean-field methodesx tendbse yontdh er estricpthiyvsei csaelt titnhga te ngendertehde m. Firsats,r eportbeyds everoaftl h ep aperisnt hivso lumteh,e rhea veb eena number ofe mpiricsaulc cessiensv olvimnega n-field metihnop drso,b lemfsa rf romt he physiscest tiSnegc.o nmda,n yo ft hea ssumptihoanvseb eeni mposewdi th tghoea l ofo btainiannagl ytirceaslu lptasr,t iculaaspr alryot f t heh untf oprh aster ansitions. Vieweads a computatiomneatlh odolomgeya,n -fietlhde ormya y notr equisruec h stronsgy mmetrioersh omogeneitTiheisr.dt ,h eries r easotno b elietvhea tt he exaccta lculattieocnh niquaensd m ean-fietledc hniqueexsp lociotm plementary aspecotfsp robabilgirsatpihci mcoadle slt ructuarnedt, h ahty britde chniqmuaeys alloswt ronign teracttioob nesr emoveuds inegx accta lculatrieovnes,a lmionrge homogeneo"urse sidutahlasct"a nb eh andlevdi am ean-fiealldg orithms. Consideratsiuocnhas st hesfeo rmt hep rincispuablj emcatt teorft heb ooka nd area ddressiendm anyo fi tsc hapteWrhsi.l et heb ookd oesa n admirabjloebo f coveritnhgeb asiocfsm ean-fietlhde oriynt hec lassisceatlt ionfIg s inagn dr elated modelst,h em aint hrusitst hed etailceodn sideraotfit ohnen ewl inkbse tween computatiaonndg enerparlo babilmiosdteilci tnhga tm ean-fiemledt hodpsr omise toe xposTeh.i si sa ne xcitianngdt imeltyo piacn,d t hec urrebnoto kp rovidtehse besttr eatmeynetta vailable. Michae1l.J ordan Berkeley
The list of books you might like
