Reinforcement Learning for Adaptive Dialogue Systems: A Data-driven Methodology for Dialogue Management and Natural Language Generation

271 Pages·2011·7.06 MB·English

by  Verena Rieser

Theory and Applications
of Natural Language Processing
Series Editors:
GraemeHirst(Textbooks)
EduardHovy(Editedvolumes)
Mark Johnson(Monographs)
A im sandScope
The field of Natural Language Processing (NLP) has expanded explosively over the past
decade: growing bodies of available data, novel fieldsof applications, emerging areas and
newconnectionstoneighboringfieldshaveallledtoincreasingoutputandtodiversification
ofresearch.
“TheoryandApplicationsofNaturalLanguageProcessing”isaseriesofvolumesdedicated
toselectedtopicsinNLPandLanguageTechnology.Itfocusesonthemostrecentadvances
inallareasofthecomputationalmodelingandprocessingofspeechandtextacrosslanguages
and domains. Due to the rapid pace of development, the diversity of approaches and
application scenarios are scattered in an ever-growing mass of conference proceedings,
making entry into the field difficult for both students and potential users. Volumes in the
seriesfacilitatethisfirststepandcanbeusedasateachingaid,advanced-level information
resourceorapointofreference.
The series encourages the submission of research monographs, contributed volumes and
surveys,lecturenotesandtextbookscoveringresearchfrontiersonallrelevanttopics,offering
aplatform for the rapid publication of cutting-edge research as well as for comprehensive
monographsthatcoverthefullrangeofresearchonspecificproblemareas.
The topics include applications of NLP techniques to gain insights into the use and
functioningoflanguage,aswellastheuseoflanguagetechnologyinapplicationsthatenable
communication,knowledgemanagementanddiscoverysuchasnaturallanguagegeneration,
informationretrieval,question-answering,machinetranslation,localizationandrelatedfields.
Thebooksareavailableinprintedandelectronic(e-book)form:
*DownloadableonyourPC,e-readeroriPad
*Enhanced by Electronic Supplementary Material, such as algorithms, demonstrations,
software,imagesandvideos
*Available online within an extensive network of academic and corporate R&D libraries
worldwide
*Neveroutofprintthankstoinnovativeprint-on-demandservices
*Competitively priced print editions for eBook customers thanks to MyCopy service
http://www.springer.com/librarians/e-content/mycopy
For further volumes:
http://www.springer.com/series/8899
Verena Rieser •Oliver Lemon
Reinforcement
Learning for Adaptive
Dialogue Systems
A Data-driven Methodology for
Dialogue Management and Natural
Language Generation
Verena Rieser Oliver Lemon
School of Mathematical School of Mathematical
and Computer Sciences and Computer Sciences
Heriot-Watt University Heriot-Watt University
Edinburgh EH14 4AS Edinburgh EH14 4AS
United Kingdom United Kingdom
[e-mail] [e-mail]
ISSN 2192-032X e-ISSN 2192-0338
ISBN 978-3-642-24941-9 e-ISBN 978-3-642-24942-6
DOI 10.1007/978-3-642-24942-6
Springer Heidelberg Dordrecht London New York
Library of Congress Control Number: 2011942326
Mathematics Subject Classification (2010): 68-XX, 68Txx, 68T05, 68T50, 68T37, 68T42
© Springer-Verlag Berlin Heidelberg 2011
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is
concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting,
reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication
or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965,
in its current version, and permission for use must always be obtained from Springer. Violations are
liable to prosecution under the German Copyright Law.
The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply,
even in the absence of a specific statement, that such names are exempt from the relevant protective laws
and regulations and therefore free for general use.

Printed on acid-free paper

Springer is part of Springer Science+Business Media (www.springer.com)
Preface
Thepastdecadehasseenarevolutioninthefieldofspokendialoguesystems.Asin
otherareasofComputerScienceandArtificialIntelligence,data-drivenmethodsare
nowbeingusedtodrivenewmethodologiesforsystemdevelopmentandevaluation.
Thesemethodsareprovingtobemorerobust,flexible,andadaptivethanthelargely
rule-basedapproacheswhichprecededthem.
Wehopethatthisbookisacontributiontothatongoingchange.Itdescribes,in
detail, a new methodology for developing spoken dialogue systems – in particular
theDialogueManagementandNaturalLanguageGenerationcomponents–which
starts with human data, and culminates in evaluation with real users. The journey
thereforestartsandendswithhumanbehaviourininteraction,andexploresmethods
for learning from the data, for building simulation environments for training and
testingsystems,andforevaluatingtheresults.
Thedetailedmaterialcovers:SpokenandMultimodaldialoguesystems,Wizard-
of-Ozdatacollection,UserSimulationmethods,ReinforcementLearning,andEval-
uationmethodologies.
Thisbookisthereforeintendedasresearchguidewhichnavigatesthroughade-
tailedcasestudyindata-drivenmethodsfordevelopmentandevaluationofspoken
dialoguesystems.Commonchallengesassociatedwiththisapproacharediscussed
andexamplesolutionsprovided,forexample,howtolearnfromlimitedamountsof
data. As such, we hope it will provide insights, lessons, and inspiration for future
researchanddevelopment–notonlyforspokendialoguesystemsinparticular,but
fordata-drivenapproachestohuman-machineinteractioningeneral.
Edinburgh, VerenaRieser
September2011 OliverLemon
v

Acknowledgements
Theresearchpresentedherewouldnothavebeenpossiblewithoutinteractionwith
ourresearchcommunity,andweowespecialthankstotheparticipantsoftheTALK
andCLASSiCprojectsfordiscussingtheideaspresentedinthisbook.
ThereisalivelyinternationalresearchcommunityinSpokenDialogueSystems
whose interest, criticism, and encouragement have helped to shape this work. We
areespeciallyindebtedtoSteveYoungandhiscolleaguesatCambridgeUniversity,
who have had a deep influence on our ideas. Members of the Interaction Lab1 at
Heriot-Watt University’s School of Mathematics and Computer Science have also
helpedtodiscussanddeveloptheseideasoverseveralyears.
InparticularwewouldalsoliketothankDr.XingkunLiu,Dr.HelenHastie,Dr.
Ivana Kruijff-Korbayova´, Dr. Tilman Becker, and other colleagues from Saarland
and Edinburgh Universities for helping with the data collection and evaluation in-
volvedinthiswork.WeespeciallythankProfessorManfredPinkalforhisguidance.
The research leading to these results has received funding from the European
Community’sSeventhFrameworkProgramme(FP7,2007-2013)undergrantagree-
ment number 216594 (“Computational Learning in Adaptive Systems for Spoken
Conversation”, CLASSiC project: www.classic-project.org), the EC FP6
project “TALK: Talk and Look, Tools for Ambient Linguistic Knowledge” (IST
507802,www.talk-project.org),fromEPSRCgrantnumbersEP/E019501/1
andEP/G069840/1,andfromtheInternationalResearchTrainingGroup“Language
Technology and Cognitive Systems”, Saarland University, funded by the German
ScienceFoundation(DFG).
Finally,thefirstauthorwouldliketothankherparentsFranzandTatjanaRieser
for their support and encouragement. The second author thanks his family for the
decadesofrewardinglearningexperienceswhichhavemadethisbookpossible.
1http://www.macs.hw.ac.uk/InteractionLab/
vii

Contents
1 Introduction................................................... 1
1.1 TheDesignProblemforSpokenDialogueSystems .............. 1
1.2 Overview ................................................. 2
1.3 StructureoftheBook ....................................... 4
PartI FundamentalConcepts
2 Background ................................................... 9
2.1 Human-ComputerInteraction ................................ 10
2.2 DialogueStrategyDevelopment .............................. 11
2.2.1 ConventionalDevelopmentLifecycle ................... 12
2.2.2 EvaluationandStrategyQualityControl................. 13
2.2.3 StrategyImplementation .............................. 17
2.2.4 ChallengesforStrategyDevelopment ................... 19
2.3 Literaturereview:LearningDialogueStrategies ................. 21
2.3.1 MachineLearningParadigms .......................... 21
2.3.2 SupervisedLearningforDialogueStrategies ............. 22
2.3.3 DialogueasDecisionMakingunderUncertainty .......... 23
2.3.4 ReinforcementLearningforDialogueStrategies .......... 24
2.4 Summary ................................................. 26
3 ReinforcementLearning ........................................ 29
3.1 TheNatureofDialogueInteraction............................ 30
3.1.1 DialogueisTemporal................................. 30
3.1.2 DialogueisDynamic ................................. 31
3.2 ReinforcementLearning-basedDialogueStrategyLearning ....... 32
3.2.1 DialogueasaMarkovDecisionProcess ................. 32
3.2.2 TheReinforcementLearningProblem................... 36
3.2.3 Model-basedvs.Simulation-basedStrategyLearning ...... 42
3.3 DialogueSimulation ........................................ 45
3.3.1 Wizard-of-OzStudies ................................ 45
ix