Bayesian Analysis in Natural Language Processing

345 Pages·2019·5.087 MB·

by  Shay Cohen, Graeme Hirst

Series ISSN 1947-4040 C
O
H
E
N

B
A
Series Editor: Graeme Hirst, University of Toronto Y
E
S
I
A
Bayesian Analysis in Natural Language Processing, Second Edition N Bayesian Analysis
A
Shay Cohen, University of Edinburgh N
A
L
Y
Natural language processing (NLP) went through a profound transformation in the mid-1980s
S
I in Natural Language
when it shifted to make heavy use of corpora and data-driven techniques to analyze language. S
I
Since then, the use of statistical techniques in NLP has evolved in several ways. One such N
N
example of evolution took place in the late 1990s or early 2000s, when full-fledged Bayesian
A
T
machinery was introduced to NLP. This Bayesian approach to NLP has come to accommodate
U Processing
various shortcomings in the frequentist approach and to enrich it, especially in the unsupervised R
A
setting, where statistical learning is done without target prediction examples. L
L
In this book, we cover the methods and algorithms that are needed to fluently read Bayesian A
N
learning papers in NLP and to do research in the area. These methods and algorithms are G
Second Edition
partially borrowed from both machine learning and statistics and are partially developed “in- U
A
house” in NLP. We cover inference techniques such as Markov chain Monte Carlo sampling and G
E
variational inference, Bayesian estimation, and nonparametric modeling. In response to rapid P
R
changes in the field, this second edition of the book includes a new chapter on representation O
learning and neural networks in the Bayesian context. We also cover fundamental concepts in C
E
Bayesian statistics such as prior distributions, conjugacy, and generative modeling. Finally, we S
S
I
review some of the fundamental modeling techniques in NLP, such as grammar modeling, N
G
neural networks and representation learning, and their use with Bayesian analysis.
, 2
N
D Shay Cohen
E
D
ABOUT SYNTHESIS
ThDigisi tvaol lLumibrea irsy ao fp Erinngteinde evreirnsgio ann odf C ao wmoprukt etrh Sacti eanpcpee. a Srsy nint htehsei sS lyenctthuerseiss . M
provide concise original presentations of important research and O
R
development topics, published quickly in digital and print formats. For
G
more information, visit our website: http://store.morganclaypool.com A
N

&

C
L
A
store.morganclaypool.com Y
P
O
O
L

Bayesian Analysis in
Natural Language Processing
Second Edition
Synthesis Lectures on Human
Language Technologies
Editor
GraemeHirst,UniversityofToronto
SynthesisLecturesonHumanLanguageTechnologiesiseditedbyGraemeHirstoftheUniversity
ofToronto.Theseriesconsistsof50-to150-pagemonographsontopicsrelatingtonatural
languageprocessing,computationallinguistics,informationretrieval,andspokenlanguage
understanding.Emphasisisonimportantnewtechniques,onnewapplications,andontopicsthat
combinetwoormoreHLTsubfields.
BayesianAnalysisinNaturalLanguageProcessing,SecondEdition
ShayCohen
2019
ArgumentationMining
ManfredStedeandJodiSchneider
2018
QualityEstimationforMachineTranslation
LuciaSpecia,CarolinaScarton,andGustavoHenriquePaetzold
2018
NaturalLanguageProcessingforSocialMedia,SecondEdition
AtefehFarzindarandDianaInkpen
2017
AutomaticTextSimplification
HoracioSaggion
2017
NeuralNetworkMethodsforNaturalLanguageProcessing
YoavGoldberg
2017
Syntax-basedStatisticalMachineTranslation
PhilipWilliams,RicoSennrich,MattPost,andPhilippKoehn
2016
iii
Domain-SensitiveTemporalTagging
JannikStrötgenandMichaelGertz
2016
LinkedLexicalKnowledgeBases:FoundationsandApplications
IrynaGurevych,JudithEckle-Kohler,andMichaelMatuschek
2016
BayesianAnalysisinNaturalLanguageProcessing
ShayCohen
2016
Metaphor:AComputationalPerspective
TonyVeale,EkaterinaShutova,andBeataBeigmanKlebanov
2016
GrammaticalInferenceforComputationalLinguistics
JeffreyHeinz,ColindelaHiguera,andMennovanZaanen
2015
AutomaticDetectionofVerbalDeception
EileenFitzpatrick,JoanBachenko,andTommasoFornaciari
2015
NaturalLanguageProcessingforSocialMedia
AtefehFarzindarandDianaInkpen
2015
SemanticSimilarityfromNaturalLanguageandOntologyAnalysis
SébastienHarispe,SylvieRanwez,StefanJanaqi,andJackyMontmain
2015
LearningtoRankforInformationRetrievalandNaturalLanguageProcessing,Second
Edition
HangLi
2014
Ontology-BasedInterpretationofNaturalLanguage
PhilippCimiano,ChristinaUnger,andJohnMcCrae
2014
AutomatedGrammaticalErrorDetectionforLanguageLearners,SecondEdition
ClaudiaLeacock,MartinChodorow,MichaelGamon,andJoelTetreault
2014
iv
WebCorpusConstruction
RolandSchäferandFelixBildhauer
2013
RecognizingTextualEntailment:ModelsandApplications
IdoDagan,DanRoth,MarkSammons,andFabioMassimoZanzotto
2013
LinguisticFundamentalsforNaturalLanguageProcessing:100Essentialsfrom
MorphologyandSyntax
EmilyM.Bender
2013
Semi-SupervisedLearningandDomainAdaptationinNaturalLanguageProcessing
AndersSøgaard
2013
SemanticRelationsBetweenNominals
ViviNastase,PreslavNakov,DiarmuidÓSéaghdha,andStanSzpakowicz
2013
ComputationalModelingofNarrative
InderjeetMani
2012
NaturalLanguageProcessingforHistoricalTexts
MichaelPiotrowski
2012
SentimentAnalysisandOpinionMining
BingLiu
2012
DiscourseProcessing
ManfredStede
2011
BitextAlignment
JörgTiedemann
2011
LinguisticStructurePrediction
NoahA.Smith
2011
LearningtoRankforInformationRetrievalandNaturalLanguageProcessing
HangLi
2011
v
ComputationalModelingofHumanLanguageAcquisition
AfraAlishahi
2010
IntroductiontoArabicNaturalLanguageProcessing
NizarY.Habash
2010
Cross-LanguageInformationRetrieval
Jian-YunNie
2010
AutomatedGrammaticalErrorDetectionforLanguageLearners
ClaudiaLeacock,MartinChodorow,MichaelGamon,andJoelTetreault
2010
Data-IntensiveTextProcessingwithMapReduce
JimmyLinandChrisDyer
2010
SemanticRoleLabeling
MarthaPalmer,DanielGildea,andNianwenXue
2010
SpokenDialogueSystems
KristiinaJokinenandMichaelMcTear
2009
IntroductiontoChineseNaturalLanguageProcessing
Kam-FaiWong,WenjieLi,RuifengXu,andZheng-shengZhang
2009
IntroductiontoLinguisticAnnotationandTextAnalytics
GrahamWilcock
2009
DependencyParsing
SandraKübler,RyanMcDonald,andJoakimNivre
2009
StatisticalLanguageModelsforInformationRetrieval
ChengXiangZhai
2008
Copyright©2019byMorgan&Claypool
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmittedin
anyformorbyanymeans—electronic,mechanical,photocopy,recording,oranyotherexceptforbriefquotations
inprintedreviews,withoutthepriorpermissionofthepublisher.
BayesianAnalysisinNaturalLanguageProcessing,SecondEdition
ShayCohen
www.morganclaypool.com
ISBN:9781681735269 paperback
ISBN:9781681735276 ebook
ISBN:9781681735290 epub
ISBN:9781681735283 hardcover
DOI10.2200/S00905ED2V01Y201903HLT041
APublicationintheMorgan&ClaypoolPublishersseries
SYNTHESISLECTURESONHUMANLANGUAGETECHNOLOGIES
Lecture#41
SeriesEditor:GraemeHirst,UniversityofToronto
SeriesISSN
Print1947-4040 Electronic1947-4059
Bayesian Analysis
in Natural Language Processing
Second Edition
Shay Cohen
UniversityofEdinburgh
SYNTHESISLECTURESONHUMANLANGUAGETECHNOLOGIES#41
M
&C Morgan &cLaypool publishers
ABSTRACT
Naturallanguageprocessing(NLP)wentthroughaprofoundtransformationinthemid-1980s
when it shifted to make heavy use of corpora and data-driven techniques to analyze language.
Sincethen,theuseofstatisticaltechniquesinNLPhasevolvedinseveralways.Onesuchexam-
pleofevolutiontookplaceinthelate1990sorearly2000s,whenfull-fledgedBayesianmachin-
erywasintroducedtoNLP.ThisBayesianapproachtoNLPhascometoaccommodatevarious
shortcomingsinthefrequentistapproachandtoenrichit,especiallyintheunsupervisedsetting,
wherestatisticallearningisdonewithouttargetpredictionexamples.
In this book, we cover the methods and algorithms that are needed to fluently read
BayesianlearningpapersinNLPandtodoresearchinthearea.Thesemethodsandalgorithms
are partially borrowed from both machine learning and statistics and are partially developed
“in-house” in NLP. We cover inference techniques such as Markov chain Monte Carlo sam-
plingandvariationalinference,Bayesianestimation,andnonparametricmodeling.Inresponse
to rapid changes in the field, this second edition of the book includes a new chapter on rep-
resentation learning and neural networks in the Bayesian context. We also cover fundamental
concepts in Bayesian statistics such as prior distributions, conjugacy, and generative modeling.
Finally, we review some of the fundamental modeling techniques in NLP, such as grammar
modeling,neuralnetworksandrepresentationlearning,andtheirusewithBayesiananalysis.
KEYWORDS
naturallanguageprocessing,computationallinguistics,Bayesianstatistics,Bayesian
NLP, statistical learning, inference in NLP, grammar modeling in NLP, neural
networks,representationlearning