Graphical models : foundations of neural computation PDF
Preview Graphical models : foundations of neural computation
GraphicMaold els CopyrightedM aterial ComputatioNneaulr oscience TerreJnS.ce ej nowasnkdTi o masAo.P oggeidoi,t ors NeuraNle tisn E lectFriiscWh a, lteHre iligenb1e9r19g , TheC omputatiBornaailPn a,t riSc.iC ah urchlaannddT errenJc.Se e jnowski, 1992 DynamiBci ologNiectawlo rTkhse:S tomatogaNsetrrviocSu yss teemd,ie td by RonalMd. H arris-WarErvieMc akr,d eArl,l eIn.S elversatnodnM ,a urice Moulin1s9,9 2 The NeurobioolfNo eguyr aNle tworekdsi,t bedyD anieGla rdne1r9,9 3 Large-Scale TNheeuorrooinfeat slh eB raiend,i tbeydC hristKoofc ha ndJ oeLl. Davis1,9 94 TheT heoretFiocuanld atoifDo enn driFtuicn ctiSoenl:e cPtaepde rosfW ilfridR ail witCho mmentareideist,be ydI daSne gevJ,o hnR inzealn,dG ordoMn. Shepher1d9,9 5 ModelosfI nformaPtrioocne ssiintn hgBe a saGla ngliead,i tbeydJ amecs. HoukJo,e Ll. D aviasn,d D aviGd. B eise1r9,9 5 SpikeEsx:p lortihneNg e llrCaold eF,r edR iekDea,v iWd arlaRnodbd ,e R uyter vanS teveninacnkd,W illiBaima le1k9,9 7 NeuronNse,t woraknsd,M otoBre havieodri,t beydP auSl. G .S teiSnt,e n GrillnAelrl,eI n.S elversatnodnD ,o uglaGs.S tuar1t9,9 7 MethoidnsN euronMaold eliFnrgo:m I ontso N etworskesc,o nedd itieodni,t ed by ChriKsotcohaf n dl danS egev1,9 98 FundamenotfaN lesu raNle twoMrokd eliNnegu:r opsychaonldoC gyog nitive Neuroscieednictebe,dy R andolpW.h ParkDsa,n ieSl.L evinaen,d D ebrLa. Long1,9 98 NeuraClo deasn dD istribRuetperde sentaFotiunodnatsoin:s o Nfe urCaolm putation, editbeydL aurenAcbeb otatn dT errenJc.Se e jnows1k9i9,7 Unsupervised LFeoaumnidnagt:io ofNn esu raClo mplltaetdiiotneb,dy G eoffrey Hintoann dT errenJc.Se e jnows1k9i9,7 FasOts cillaitniC oonrst iCciarlc uiRtosg,eD r. T raubJ,o hn G.R .J efferaynsd, MileAs. Wh ittingt1o9n9,9 ComputatiVoinsailolI nzjo:r matiPorno cessiinPne gr pcteioannd VisuBaelh avior, HanspetAe.Mr all2o0t00, GraphiMcoadle lFso:u ndatioofNn es urCoamlpll tation, editbeydM ichaIe.l Jordaann dT errenJcS.ee jnows2k0i0,1 Self-OrganMiaziJFn'og r matiFoonu:n datioofNne lsr lalC omp"tateidoint,eb dy KlaOubse rymearan dT errenJc.Se e jnows2k0i0,1 Copyrighted Material GraphicMaold elsF:o undatioonfNs e uraClo mputation Editebdy M ichaeIl.J ordaann dT errenJc.Se e jnowski A BradfBoookr d TheM ITP rses CambridMagseasc,h� pyrlgh tedM aterila LondEonng,l and V 200M1a ssachusIentsttsi toufTt eec hnology Alrli ghrtess ervNeod p.a rotf t hibso okm ayb er eproducine adn yf ormb ya nye lectronic orm echanicmaela ns (includpihnogt ocopyirnegc,o rdionrgin ,f ormations toraagne d retriewviatlh)o upte rmissiionwn r itifnrgo mt hep ublisher. Thisb ookw ass eti nP alatianndo p rintaendd b oundi nt heU niteSdt atoefAs m erica. LibraorfyC ongresCsa taloging-in-PuDbaltiac ation Graphicmaold els; nfdoautioonfsn eurcaolm puta/te idoint beydM ichaIeJ.l o rdan andT errenJc.See jnokwis. p. cm.- (AB radofrd book()C omputatinoenaulr oscience) ISBN0 -262-600(4p2b-k0:. a lkp.a per) 1.Ne uranle twork(sC ompustceire nc2.eC )o mputgrearp hicsI.J. o rdaMni,c hael Irwin1,9 56-IISe.jn owskTie, rreJn.(c Tee rreJnocsee pIhI)SeI ri.es .IV .B radbfoookr d QA76.8.7G 72 2001 006.3'2-dc21 2001030212 CopyrighMtaetde rial Contents SeriFeosr eword vii Sources ix Introduction xi 1 Probabilistic IndependfeonrHc ied dNeeMnta wrokrokvs ProbabiMloidteyl s 1 PadhraSimcy thD,a viHde ckermaann,dM ichaIe.Jl o rdan 2 Learning andR elearningi nB oltzmaMnna chines 45 G.E .H intoann dT .J S.e jnowski 3 Learning inB oltzmaTnnre es 77 LawrenScaeu aln dM ichaIe.Jord/ml 4 Deterministic Botzmanln Learning PerforSmtse epeDsets ceinnt Weight-Space 89 GeoffreEy. H inton 5 Attractor DyinnaF meiecdsf orwNaerudr aNle tworks 97 LawrencK.e S aualn dM ichaIe.Jl o rdan 6 EfficienLte arning inB oltzmaMnna chisnU esinLgi neaRre sponse Theory 121 H.J .K appeann dF .B .R odriguez 7 AsymmetrPiacr alBloellt zmaMnna chinAree s BeliNeeft works 141 RadforMd. N eal 8 VariatiLoenaarninlg inN onlineaGra ussiBaenl iNeeftw orks 145 BrendJa.Fn r eayn dG eoffrey E.H inton 9 MixturoefsP robabilPirsitnicci Cpoamlp onenAtn alyzers 167 MichaEe.lT ippianngdC hristoMp.hB eirs hop 10I ndependFeanctt Aonra lysis 207 H.A ttias 11H ierarchMixituresc al ofE xperatnsd t heE M Algorithm 257 Micha1e.Jl o rdaannd R obeAr.tJ acobs 12H iddeNne uraNle tworks 291 AndersK rogha ndS BTenK amarRiici s 13V ariatiLoenaarningl foSrw itchiSntga te-SMpoadceel s 315 ZoubiGnh ahramaanndiG eoffreEy. H inton 14N onlineTairm e-SerPireesd ictwiiotnhM issianngd N oisDya ta 349 VolkTerre sapn dR eimaHro fmann 15C orrectnoefLs osc aPlr obabiPlriotpya gatiinGo rna phicMaold els withL oops 367 YairW eiss Index Copyrighted Material 409 SerieFso reword Computationneaulr osciiesna cnae p proactho u nderstantdhiein ngf or matiocno nteonftn eurasli gnablys m odelintgh en ervoussy steamt manyd ifferesnttr uctusrcaall eisn,c luditnhgeb iophysictahlec, i rcuit, andt hes ystemlse velCso.m putesri mulatioofnn esu ronasn dn eural networakrse c omplementtaotr rya dititoencahnli quiensn euroscience. This books eriweesl comecso ntributtihoanltsi ntkh eoretsitcuadli es withe xperimeanptparlo acthoeu sn derstanidnifnogr matpiroonc essing in then ervoussy steAmr.e aasn dt opiocfsp aritculianrt eriensctl ude biophysimceaclh anisfmosrc omputatiinon ne uroncso,m puter simu latioonfns e uracli rcumiotdse,l osf l earnirnepgr,e sentaotfis oenn sory informatiinno enu ranle tworkssy,s temmosd elosf s ensory-mionttoer gratiaonnd,c omputatiaonnaally soifps r obleimnsb iologisceanls ing, motocro ntraonld,p erception. TerrenJc.Sejen owski TomasAo. P oggio CopyrighteMadt erial Sources SmythP,. ,H eckermaDn.,,a ndJ ordaMn., I .1 997Pr.ob abiliisntdiecp endence networfkosr dhdienM arkovp robabimloidteyl sN.e uraClo mputat9i(o2n) , 227-269. HintoGn.,E .a,n dS ejnowsTk.Ji .,1 98L6e.a rninagn dr elearniinnBg o ltzmann machinesIn. P arallDeils tribPurtoecde ssEixnpgl:o raitnit ohnMesi crostruofc ture CognitiVoonl,u m1e:F oundatiDo.En .sR ,u meIhaarntdJ .L .M cClell,ae ndds., pp.2 82-3M1I7T.P ressC,a mbridge. SaulL,.and , JordaMn.,1 .19 9L4e.arnin g in Boltzmatrennes . NeurCaolm putation 6(6),1174-1184. Hint,oG n.E .1 98D9e.t erminiBsotlitcz malnena rninpge rforsmtse epedsets cent inw eight-spNaecuera.lC omputati1o(n,1 1 )42-150. SaulL,.K .,a ndJ ordaMn., I .2 000.A ttracdtyonra micins f eedforwanredu ral networkNse.u ralC omputati1o2n( 61)3,1 33-51.3 KappenH,. J .a,n dF .B .R odrigue1z9.9 8E.ffic ienlte aming inB oltzmann machinesu singl inearre spontshee orNye.u rCaolm putati1o0n( 51)31,7 -1156. NealR,. 1 992A.sy mmetrpiacr alBloellt zmamnna chineasr eb elineeft works. NeurCaolm putat4i(o6n8) 3,2 -834. Fre,yB .J .a,n dH intoGn.,E .1 99V9a.r latiloenarninagl in nonlinGeaaurs sian belineeft workNse.u raClo mputat1i1o(n11 )9,32 -13. TippinMg.,E ., aBnids hoCp.,M . 199M9i.x turoefps r obabilpirsitniccic poaml ponenatn alyzeNresu.r aClo mputat1i1o(n244 )3-,4 82. AttiaHs.1, 99I9n.d ependfeanctt aonra lysNiesu.r Caolm putat1i1o(n48 )0,3 -18.5 JordaMn.,I .a,n dJ acobRs., A1.9 9H4i.e rarchmiicxatlu roefes x peratnsdth e EM algoritNhemu.r al Compu6t(a21t)i8,o12 n-1 4. KroghA,. , aRnidi sS,.K .1 999H.id denne uranle tworksN.e uraClo mputation 11(2),541-563. Ghahramazn.ia,,n dH intoGn.,E .2 00V0a.r iatiloenaamlifnogsr w itchsitnagt e spacmeo delNse.u raClo mputat1i2o(n48 )3-,18 64. Tresp, V.,a ndH ofmann, R.1 998No.n lineatri me-seprrieedsi ctwiiothn m issing andn oisdya taN.e uralC omputat1i0o(n37 )3,1 -747. WeissY,.2 00C0o.r rectneosfls o caplr obabiplriotpya gatini gorna phical models witlho opNse.ur aClo mputat1i2(o1n1) -,4 1. CopyrighteMda terial Introduction A" graphmiocdaelil sa t ypoefp robabnieltiwsottrihkcah tasr ooitns II several diffecroemnmtu nrieitsniecealsru,cad rhit nigfii cnitaell ligence (Pela1 r988s)t,a ti(sLtaiucrsi1 t9z9e6an)n ,dn eurnaelt wo(rHkesr tz, Krogahn,dP alm1e9r9 1T)h.eg raphmiocdaelfl rsa mewporrokv iad es clemaant hemaftoircmaallt ihsahmta msa dietp ossitboul ned erstand three latioanmsohnaigw p isd vea rioefnt eyt work-abpapsreodat coh es computaatnidio,nnp artictuoul nadre,r smtaannynd e urnaelt work algoriatnhdam rsc hiteacsit nusrteason fca e bsr oadperro babilistic methodoMloorgeyo.vt ehrif,so rmfarla mewhoarmska diet p ossible toi denttihoffsyee a tuorfne esu rnaelt woarlkg Oriatnhdam rsc hitec turtehsaa trn eo vaenldt oe xtetnhdet moo thmeorr gee negrraalp hical modeTlhsi.is n terbpeltawyet ehgnee nefroarlmf aarlm ewoorfgk r aph icmaold ealnsdt heex ploroafnt eiwoa nl goriatnhdam rsc hiteicst ures exempliinft ihecedh apters iintn hcilvsuo dleudmT eh.e sceh apters, chosferno Nme urCaolm putatiionnc,l muadnefy o undatpiaopneoarfls histoirmipcoarlta aswn ecleal sp apetrhsaa tra ett hree seafrrcohn tier. Thev oluimsie n tenfdoearbd r oarda nogfes tudernetsse,a racnhde rs, practitwihooan reiern st eriensu tnedde rstatnhbdeai snpigrc i nciples underlgyrianpgh icala nidmn oa dplepylisnt gh etmop ractpircoabll ems. Probabialnidis ntfiocr mationa-ptphreooarhceahtveibesce come dominiantn htne e urnaelt wolrikt eraasrt eusreea,r hcahvaeetr tse mpted tof ormalize "iandn aeputrciaovlmi ptuyt"aa tniduo nnd erswthayn d somaed aptmievteh opdesr fobremt ttehrao nt heThres p.r obabilistic framewhoaraskl spor oviadg eudi idnet heex ploroafnt eiwoa nl go rithamnsda rchiteActtt husera emste.i mteh,ne o tioo"fnl ocahlaist y" contitnoue exdea rk te cyo nstornan ienutrn aelt worreks earrecsht,r ict intgh kei nodfsa rchiteacntdau lrgeosr itthhaamtrss e t udThiee dp.ar ticular orfeg lreavpahmnioccdeae ltl ots h irse seaerffcoishr tth atth e graphmiocdaelfl rsa mewporrokv ifdoersmd aelfi nitoifbo ontashd ap tivaintdyl ocalIitdt oyes.so b yf orgai mnagt hemaltiinbckea tlw een probabtihleioatrnyydg rapthhe ory. Graphmiocdaelul ssge r apthors e preasnedmn atn ipujloainpttre o b abildiitsyt ribTuhtegi roanpushn. d erlayg irnagp hmiocdaemlla yb e direcitnwe hdi,cc ha stehm eo deilos f treenf etroar seab d e lief neotr work aB ayilelsnn etwoorrkt, h ger apmha yb eu ndireicnwt heidc,ch a steh e modeilsg enerraelfleyrt roae sda M arkorva ndofime ldA. g raphical modehla sb otah s trucctoumrpaoln ent-ebnyct ohdpeea dt toefrn edgine st hger aph-aap nadr amectormipco nent-ebnync uomdeerd ic"aplotentaisaslosc"wi iattshee dot fse dgienst hger apThh.re e lation shibpe twetehnec soem ponuenndtesr tlhiceeo sm putamtaicohniaynl e r associwaittgehrd a plhm iocdaelIsnp .a rticguelnaeirrfern,ae ln caegl o rithamls lsotwa ti,Q!lsatnitcja!'(li£� ��.. ! 2:!�l ikeliahnodco odnsd itional probabialniidtn ifeosr)h1. \:l(l��WIl�diftti(essua csmh u tuinaflo r- ( Introduction xiii (chapte1r) w,h icph roviad es horotv erviwe.A fulplr esentactainbo en founind anyo fs everrecenal tt extbook(se .gC.o,w elle ta 11. 999Se)e. alsJoo rda(n1 99f9or)s e vertault orairatli ctlheaspt r ovibdaes ibca ck grounfdo rth e chapteprrse senhteerde . TheB oltzmannM achien ___ ___ _________ _ TheB oltzmmJlJlncnh (icnhea pt2e)ir s a probabilniesttwiocro kfb inary nodesH.i storictahleBl oyl,t zmamnna chinpel ayeadn i mportarnotli en thed evelopmoefnt th en euranle tworfike lda,st hefi rsgte nermaull ti layearr chitecttoeu mrpel ohyi ddeunn itbse tweent hei npuatn do utput nodesA.� w e discuisnts h isse ctitohneB, o ltzmamnna chiniesa special casoef a nun directgerda phimcoadle lo,r M arkov randofime l(dM RF). Thei nfereanncdel earnianlgg oritfhomrBs o ltzmamnna chineasn,di n particuthel atrr eatmeonfht i ddeunints , exemplimfyo reg enersaoll u tiontsot hep robleomfin ferenacnde learninigng raphimcoadle lwsi th latevnatr iables. Whileg eneraMlR F'sr epresejnoti nptr obabildiitsyt ributaiso ns producotfsa rbitrlaorcyaf lu ncti(o"npso tention athles "c)l iquoefts h e grapht,hle B oltzmann macahdionpet as r estripcatrem;de terizaint ion which thep otentiaarlefs o rmefdr omp airwfaicsteoT rhse.s pea irwise factotrask teh ef orme xp{JijSjwShje}r,he j i st hew eighotn thee dge betweeunni t ain dj ,a ndS ja ndS ja ret he( binarvya)l ueosfu nitis andj ,r especti(vIneal yg.e nerMaRFl, higher-oirndteerr actsiuocnhs ash jlcSjSwjoSuIcl db ei ncluded-whtehnen odeSsi S,j a,n dS ica rei na cliqunea,m elya,r em utualinltye rconnectTeadk)in.g p roducotfsth ese locaplo tentialysi eltdhse t otaplo tenteixapl{ Ejih<jSiSwjh}i,c hw,h en normalizdeedfi,n eas B oltzmadnni stribution: -E(S) e P(S=)z- (1) fora quadrateince rfugyn ctiEo(nS";t) - Ei<jh jSjSFjr'o mthis joint probabidliisttyr ibuwtei coannd, e finaer bitrcaornyd itipornoabla bilities ofo nes eotf n odegsi veann othseerto fn odesC.a lculatthiensgce o ndi tionadlesfi netsh ei nferepnrcoebf loeBrmo l tzmamnna chines. Forg enerBaoltzmanl n machineisnp, a rticfuoltrah re fu llcyo nnected Boltzmamnna chintehsa hta veg enerabelenly s tudiine dt hel iterature, theraer en os tructuprroapertiesl (conditiionndaelpd eenncietso)t ake advantaogfe,a ndt hei nferepnrcoeb leims i ntractaAbplper.o ximate inferetnecceh niqhuaevseg enerableleyne mployed-pianr ticusltaor, chastsiacm pli(nGgi bbssa mplinegn)h ancweidt hs imulataendn ealing. Althougthhe sem ethoddso providae w ay tos tudtyh eB oltzmann 1 cAl iisq afu ulel cony nected subgraAph c.l iquconsiseti ng ofn b inanodreys canb e in oneo f2 "c onfiguratiownhse,r ec ao nfiguisraa nta sisoingn moenft bai narvya lutoe eacnhod e in the cliquAe po. �ril';,-,i wra3lthWa t .aSllignas n onnegatrievanelu mber toe accho nfigurna.ti o opyng, e Matenal xiv Introduction machienmep iritchaelaylry se,l oawn da rgee nervailelewyda sc omplxe (particwuhleanur sleyid n t hes ettoifnl ge arnianlggr toihms,w here multispilmeu lated apnansesaaerlsrei e nqgu irHeidos)rtc.ia yl,lw hen them ultilayer pbeerccaempepop turl,oaB nro ltzmmaancnih ensl ost theilru ster. Thef actth atth ew orst-fcualscleoy,n necBtoeldt nznmm aacihne presennoto sp portufnoifrta isietns f erdeonecnseo ti mpltyht aB o�zl . mann machiniengs e nerparle sennost u cohp portunThiitspi oemst. ofv iewwa se mphasibzySe adu aln dJ ord(acnh ap3t)we,hr os tudied Boltzmmaancnh inienws h icthh eh idnd eunitfso rma treThee.y shoewdt haitns ucahr chiteictit snu ortne esc esstaorr eyst ot roG ibbs sampltions go ltvheie n ferpernocbel reamt;h aes ri,m pdleet erministic recurksnioownan s d ecimactainbo een m ployteocd a lcutlhaceto en di tionparlo babiTlhiett iimerese .q uifroetrdh ec omputaitspi roonp or tiontaotl h wei dtohft hger apThh.e saerB eo ltzmmaancnh itnheasct a n be" slove"d . Thed ecimartuilocena nb eg eneralbiezyeodnt dh pea irwiinstee r actitohnasct h aracttehrceil zaes sBioclmatalzn nm achiyniee,l dainn g exaccatl culmaettihoofndo g re neMrRalF sI.n terestthiirnsug llieysa , specciaasloe ft hjeu ncttiroemnee thodotlhoaghtya sb eedne veloped foirn ferienna creb itgrraarpyh miocdael(l Cso weeltal l 1.9 99T)h.e re appeatrobs en op articaudlvaarn ttaotg hede e cimaatpiporno aacnhd, indetehdje u ncttiroeanep prohaacsht haed vantoafpg reo vidainn g explmiectihtof oder s timatthiteni egmc ompleoxfii ntfye renctei-mteh e compleixsei xtpyo neinntt ihaseli zoeft hlea rgesti nct lhtierq iuaen gu latgerda opflt lh nee twoTrhku.is ti sp ossitboil dee ntsiyfsyt ematically thcel asosfBe osl tzmmaancnh ifnoewrsh icehx aicntf eriesen fficcei ent. MeanF ielAdp proximat_i___o__n__ ________ _ In1 98P7e,t erasnodAn n ders(o1n9 8pr7e)s eannta eldt ernaatpipvreho ac toinf erenfcoetr h Beo ltzmmaancnh itnheah ta hsa ds ubstainmtpiaaclt . Theiaprp rowaacshb aseodna na pproximkantoiwoinnn p hysiacss the" meafine lda"p proximaUntdieotrnh .ia sp proximtahtei( oanp, proximmaetaen)v aluoeft hec onditional dpirsotribbaubtaiitlo int y eacnho diesw ritatsea nf unctoifto hn(e a pproximmeaatnve a)l uoefs itnse ighbaonrdsa , s o-caslellefd- consseiots ftm eenatn values is obtaibnyei dt erativelyt heevsfaeul nucattiiFnoognr ts h.eB oltzmann machitnhee,si et erative teuqmou uattt ioto anksae s impcllea ssical forimn w hiceha cnho dev'asl uiest hleo gifsutnicct oifao w ne ighted sumo fi tnse ighbvoarlsu'Te hse.sa ert eh set andnaornidln aere quations proposbeyHd o pfie(l1d9 8f4o)tr h e" continHuoopfiuesl nde two"r k, andt hecya bne s hown-vLiyaa puntohve oryb-elt ooc aclolnyv ergent (CohaenndG rossb1e9r8gH3 o;p fie1l9d8 4). PeterasnodAn n dersiodne'hasa sb eetna keinnt wo somewahtd if ferendti rec.tO inoenl sinoefr eseahracsfh o cusoendo ptiimztaion problewmhse,rt eh �e& f,IW���riBf sg iverni steoa general
The list of books you might like
