Table Of Content

Series ISSN 1947-4040 C O H E N B A Series Editor: Graeme Hirst, University of Toronto Y E S I A Bayesian Analysis in Natural Language Processing, Second Edition N Bayesian Analysis A Shay Cohen, University of Edinburgh N A L Y Natural language processing (NLP) went through a profound transformation in the mid-1980s S I in Natural Language when it shifted to make heavy use of corpora and data-driven techniques to analyze language. S I Since then, the use of statistical techniques in NLP has evolved in several ways. One such N N example of evolution took place in the late 1990s or early 2000s, when full-fledged Bayesian A T machinery was introduced to NLP. This Bayesian approach to NLP has come to accommodate U Processing various shortcomings in the frequentist approach and to enrich it, especially in the unsupervised R A setting, where statistical learning is done without target prediction examples. L L In this book, we cover the methods and algorithms that are needed to fluently read Bayesian A N learning papers in NLP and to do research in the area. These methods and algorithms are G Second Edition partially borrowed from both machine learning and statistics and are partially developed “in- U A house” in NLP. We cover inference techniques such as Markov chain Monte Carlo sampling and G E variational inference, Bayesian estimation, and nonparametric modeling. In response to rapid P R changes in the field, this second edition of the book includes a new chapter on representation O learning and neural networks in the Bayesian context. We also cover fundamental concepts in C E Bayesian statistics such as prior distributions, conjugacy, and generative modeling. Finally, we S S I review some of the fundamental modeling techniques in NLP, such as grammar modeling, N G neural networks and representation learning, and their use with Bayesian analysis. , 2 N D Shay Cohen E D ABOUT SYNTHESIS ThDigisi tvaol lLumibrea irsy ao fp Erinngteinde evreirnsgio ann odf C ao wmoprukt etrh Sacti eanpcpee. a Srsy nint htehsei sS lyenctthuerseiss . M provide concise original presentations of important research and O R development topics, published quickly in digital and print formats. For G more information, visit our website: http://store.morganclaypool.com A N & C L A store.morganclaypool.com Y P O O L Bayesian Analysis in Natural Language Processing Second Edition Synthesis Lectures on Human Language Technologies Editor GraemeHirst,UniversityofToronto SynthesisLecturesonHumanLanguageTechnologiesiseditedbyGraemeHirstoftheUniversity ofToronto.Theseriesconsistsof50-to150-pagemonographsontopicsrelatingtonatural languageprocessing,computationallinguistics,informationretrieval,andspokenlanguage understanding.Emphasisisonimportantnewtechniques,onnewapplications,andontopicsthat combinetwoormoreHLTsubfields. BayesianAnalysisinNaturalLanguageProcessing,SecondEdition ShayCohen 2019 ArgumentationMining ManfredStedeandJodiSchneider 2018 QualityEstimationforMachineTranslation LuciaSpecia,CarolinaScarton,andGustavoHenriquePaetzold 2018 NaturalLanguageProcessingforSocialMedia,SecondEdition AtefehFarzindarandDianaInkpen 2017 AutomaticTextSimplification HoracioSaggion 2017 NeuralNetworkMethodsforNaturalLanguageProcessing YoavGoldberg 2017 Syntax-basedStatisticalMachineTranslation PhilipWilliams,RicoSennrich,MattPost,andPhilippKoehn 2016 iii Domain-SensitiveTemporalTagging JannikStrötgenandMichaelGertz 2016 LinkedLexicalKnowledgeBases:FoundationsandApplications IrynaGurevych,JudithEckle-Kohler,andMichaelMatuschek 2016 BayesianAnalysisinNaturalLanguageProcessing ShayCohen 2016 Metaphor:AComputationalPerspective TonyVeale,EkaterinaShutova,andBeataBeigmanKlebanov 2016 GrammaticalInferenceforComputationalLinguistics JeffreyHeinz,ColindelaHiguera,andMennovanZaanen 2015 AutomaticDetectionofVerbalDeception EileenFitzpatrick,JoanBachenko,andTommasoFornaciari 2015 NaturalLanguageProcessingforSocialMedia AtefehFarzindarandDianaInkpen 2015 SemanticSimilarityfromNaturalLanguageandOntologyAnalysis SébastienHarispe,SylvieRanwez,StefanJanaqi,andJackyMontmain 2015 LearningtoRankforInformationRetrievalandNaturalLanguageProcessing,Second Edition HangLi 2014 Ontology-BasedInterpretationofNaturalLanguage PhilippCimiano,ChristinaUnger,andJohnMcCrae 2014 AutomatedGrammaticalErrorDetectionforLanguageLearners,SecondEdition ClaudiaLeacock,MartinChodorow,MichaelGamon,andJoelTetreault 2014 iv WebCorpusConstruction RolandSchäferandFelixBildhauer 2013 RecognizingTextualEntailment:ModelsandApplications IdoDagan,DanRoth,MarkSammons,andFabioMassimoZanzotto 2013 LinguisticFundamentalsforNaturalLanguageProcessing:100Essentialsfrom MorphologyandSyntax EmilyM.Bender 2013 Semi-SupervisedLearningandDomainAdaptationinNaturalLanguageProcessing AndersSøgaard 2013 SemanticRelationsBetweenNominals ViviNastase,PreslavNakov,DiarmuidÓSéaghdha,andStanSzpakowicz 2013 ComputationalModelingofNarrative InderjeetMani 2012 NaturalLanguageProcessingforHistoricalTexts MichaelPiotrowski 2012 SentimentAnalysisandOpinionMining BingLiu 2012 DiscourseProcessing ManfredStede 2011 BitextAlignment JörgTiedemann 2011 LinguisticStructurePrediction NoahA.Smith 2011 LearningtoRankforInformationRetrievalandNaturalLanguageProcessing HangLi 2011 v ComputationalModelingofHumanLanguageAcquisition AfraAlishahi 2010 IntroductiontoArabicNaturalLanguageProcessing NizarY.Habash 2010 Cross-LanguageInformationRetrieval Jian-YunNie 2010 AutomatedGrammaticalErrorDetectionforLanguageLearners ClaudiaLeacock,MartinChodorow,MichaelGamon,andJoelTetreault 2010 Data-IntensiveTextProcessingwithMapReduce JimmyLinandChrisDyer 2010 SemanticRoleLabeling MarthaPalmer,DanielGildea,andNianwenXue 2010 SpokenDialogueSystems KristiinaJokinenandMichaelMcTear 2009 IntroductiontoChineseNaturalLanguageProcessing Kam-FaiWong,WenjieLi,RuifengXu,andZheng-shengZhang 2009 IntroductiontoLinguisticAnnotationandTextAnalytics GrahamWilcock 2009 DependencyParsing SandraKübler,RyanMcDonald,andJoakimNivre 2009 StatisticalLanguageModelsforInformationRetrieval ChengXiangZhai 2008 Copyright©2019byMorgan&Claypool Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmittedin anyformorbyanymeans—electronic,mechanical,photocopy,recording,oranyotherexceptforbriefquotations inprintedreviews,withoutthepriorpermissionofthepublisher. BayesianAnalysisinNaturalLanguageProcessing,SecondEdition ShayCohen www.morganclaypool.com ISBN:9781681735269 paperback ISBN:9781681735276 ebook ISBN:9781681735290 epub ISBN:9781681735283 hardcover DOI10.2200/S00905ED2V01Y201903HLT041 APublicationintheMorgan&ClaypoolPublishersseries SYNTHESISLECTURESONHUMANLANGUAGETECHNOLOGIES Lecture#41 SeriesEditor:GraemeHirst,UniversityofToronto SeriesISSN Print1947-4040 Electronic1947-4059 Bayesian Analysis in Natural Language Processing Second Edition Shay Cohen UniversityofEdinburgh SYNTHESISLECTURESONHUMANLANGUAGETECHNOLOGIES#41 M &C Morgan &cLaypool publishers ABSTRACT Naturallanguageprocessing(NLP)wentthroughaprofoundtransformationinthemid-1980s when it shifted to make heavy use of corpora and data-driven techniques to analyze language. Sincethen,theuseofstatisticaltechniquesinNLPhasevolvedinseveralways.Onesuchexam- pleofevolutiontookplaceinthelate1990sorearly2000s,whenfull-fledgedBayesianmachin- erywasintroducedtoNLP.ThisBayesianapproachtoNLPhascometoaccommodatevarious shortcomingsinthefrequentistapproachandtoenrichit,especiallyintheunsupervisedsetting, wherestatisticallearningisdonewithouttargetpredictionexamples. In this book, we cover the methods and algorithms that are needed to fluently read BayesianlearningpapersinNLPandtodoresearchinthearea.Thesemethodsandalgorithms are partially borrowed from both machine learning and statistics and are partially developed “in-house” in NLP. We cover inference techniques such as Markov chain Monte Carlo sam- plingandvariationalinference,Bayesianestimation,andnonparametricmodeling.Inresponse to rapid changes in the field, this second edition of the book includes a new chapter on representation learning and neural networks in the Bayesian context. We also cover fundamental concepts in Bayesian statistics such as prior distributions, conjugacy, and generative modeling. Finally, we review some of the fundamental modeling techniques in NLP, such as grammar modeling,neuralnetworksandrepresentationlearning,andtheirusewithBayesiananalysis. KEYWORDS naturallanguageprocessing,computationallinguistics,Bayesianstatistics,Bayesian NLP, statistical learning, inference in NLP, grammar modeling in NLP, neural networks,representationlearning