Man-Machine Dialogue

220 Pages·2013·1.701 MB·English

by  Frédéric Landragin(auth.)

ManMachine Dialogue
ManMachine
Dialogue
Design and Challenges
Frédéric Landragin
Series Editor
Patrick Paroubek
Firstpublished2013inGreatBritainandtheUnitedStatesbyISTELtdandJohnWiley&Sons,Inc.
Apart from any fair dealing for the purposes of research or private study, or criticism or review, as
permittedundertheCopyright,DesignsandPatentsAct1988,thispublicationmayonlybereproduced,
storedortransmitted,inanyformorbyanymeans,withthepriorpermissioninwritingofthepublishers,
or in the case of reprographic reproduction in accordance with the terms and licenses issued by the
CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the
undermentionedaddress:
ISTELtd JohnWiley&Sons,Inc.
27-37StGeorge’sRoad 111RiverStreet
LondonSW194EU Hoboken,NJ07030
UK USA
www.iste.co.uk www.wiley.com
©ISTELtd2013
TherightsofFrédéricLandragintobeidentifiedastheauthorofthisworkhavebeenassertedbyhimin
accordancewiththeCopyright,DesignsandPatentsAct1988.
LibraryofCongressControlNumber: 2013939407
BritishLibraryCataloguing-in-PublicationData
ACIPrecordforthisbookisavailablefromtheBritishLibrary
ISBN:978-1-84821-457-6
PrintedandboundinGreatBritainbyCPIGroup(UK)Ltd.,Croydon,SurreyCR04YY
Table of Contents
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
PART 1. HISTORICAL AND METHODOLOGICAL LANDMARKS . . 1
Chapter1.AnAssessmentoftheEvolution
ofResearchandSystems . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.Afewessentialhistoricallandmarks . . . . . . . . . . . . . . . . 5
1.1.1.Firstmotivations,firstwrittensystems . . . . . . . . . . . . 6
1.1.2.Firstoralandmultimodalsystems . . . . . . . . . . . . . . . 11
1.1.3.Currentsystems: multiplicityoffieldsandtechniques . . . 14
1.2.Alistofpossibleabilitiesforacurrentsystem . . . . . . . . . . . 16
1.2.1.Recordingdevicesandtheiruse . . . . . . . . . . . . . . . . 17
1.2.2.Analysisandreasoningabilities . . . . . . . . . . . . . . . . 20
1.2.3.Systemreactiontypesandtheirmanifestation . . . . . . . . 22
1.3.Thecurrentchallenges . . . . . . . . . . . . . . . . . . . . . . . . 23
1.3.1.Adaptingandintegratingexistingtheories . . . . . . . . . . 23
1.3.2.Diversifyingsystems’abilities . . . . . . . . . . . . . . . . . 25
1.3.3.Rationalizingthedesign . . . . . . . . . . . . . . . . . . . . 26
1.3.4.Facilitatingtheimplementation . . . . . . . . . . . . . . . . 27
1.4.Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Chapter2.Man–MachineDialogueFields . . . . . . . . . . . . . . . 29
2.1.Cognitiveaspects . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.1.1.Perception,attentionandmemory . . . . . . . . . . . . . . . 32
vi Man–MachineDialogue
2.1.2.Representationandreasoning . . . . . . . . . . . . . . . . . 35
2.1.3.Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.2.Linguisticaspects . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.2.1.Levelsoflanguageanalysis . . . . . . . . . . . . . . . . . . 41
2.2.2.Automaticprocessing . . . . . . . . . . . . . . . . . . . . . . 44
2.3.Computeraspects . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
2.3.1.Datastructuresanddigitalresources . . . . . . . . . . . . . 45
2.3.2.Man–machineinterfaces,plasticinterfaces
andergonomics . . . . . . . . . . . . . . . . . . . . . . . . . 46
2.4.Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Chapter3.TheDevelopmentStagesofaDialogueSystem . . . . . . 47
3.1.Comparingafewdevelopmentprogresses . . . . . . . . . . . . . 48
3.1.1.Ascenariomatchingthe1980s . . . . . . . . . . . . . . . . 48
3.1.2.Ascenariomatchingthe2000s . . . . . . . . . . . . . . . . 49
3.1.3.Ascenariotoday . . . . . . . . . . . . . . . . . . . . . . . . 51
3.2.Descriptionofthemainstagesofdevelopment . . . . . . . . . . 52
3.2.1.Specifyingthesystem’staskandroles . . . . . . . . . . . . 52
3.2.2.Specifyingcoveredphenomena . . . . . . . . . . . . . . . . 54
3.2.3.Carryingoutexperimentsandcorpusstudies. . . . . . . . . 55
3.2.4.Specifyingtheprocessingprocesses . . . . . . . . . . . . . 58
3.2.5.Resourcewritinganddevelopment . . . . . . . . . . . . . . 59
3.2.6.Assessmentandscalability . . . . . . . . . . . . . . . . . . . 61
3.3.Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Chapter4.ReusableSystemArchitectures . . . . . . . . . . . . . . . 63
4.1.Run-timearchitectures . . . . . . . . . . . . . . . . . . . . . . . . 64
4.1.1.Alistofmodulesandresources . . . . . . . . . . . . . . . . 64
4.1.2.Theprocessflow . . . . . . . . . . . . . . . . . . . . . . . . 66
4.1.3.Moduleinteractionlanguage . . . . . . . . . . . . . . . . . . 68
4.2.Design-timearchitectures . . . . . . . . . . . . . . . . . . . . . . 69
4.2.1.Toolkits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.2.2.Middlewareforman–machineinteraction . . . . . . . . . . 71
4.2.3.Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
4.3.Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
TableofContents vii
PART 2. INPUTS PROCESSING . . . . . . . . . . . . . . . . . . . . . . 75
Chapter5.SemanticAnalysesandRepresentations . . . . . . . . . . 77
5.1.Languageindialogueandinman–machinedialogue . . . . . . . 78
5.1.1.Themaincharacteristicsofnaturallanguage . . . . . . . . . 78
5.1.2.Oralandwrittenlanguages . . . . . . . . . . . . . . . . . . . 82
5.1.3.Languageandspontaneousdialogue . . . . . . . . . . . . . 83
5.1.4.Languageandconversationalgestures . . . . . . . . . . . . 84
5.2.Computationalprocesses: fromthesignaltothemeaning . . . . 85
5.2.1.Syntacticanalyses . . . . . . . . . . . . . . . . . . . . . . . . 86
5.2.2.Semanticandconceptualresources . . . . . . . . . . . . . . 87
5.2.3.Semanticanalyses . . . . . . . . . . . . . . . . . . . . . . . . 89
5.3.Enrichingmeaningrepresentation . . . . . . . . . . . . . . . . . . 91
5.3.1.Attheleveloflinguisticutterance . . . . . . . . . . . . . . . 91
5.3.2.Atthelevelofmultimodalutterance . . . . . . . . . . . . . 93
5.4.Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
Chapter6.ReferenceResolution . . . . . . . . . . . . . . . . . . . . . 95
6.1.Objectreferenceresolution . . . . . . . . . . . . . . . . . . . . . 96
6.1.1.Multimodalreferencedomains . . . . . . . . . . . . . . . . 98
6.1.2.Visualsceneanalysis . . . . . . . . . . . . . . . . . . . . . . 100
6.1.3.Pointinggestureanalysis . . . . . . . . . . . . . . . . . . . . 101
6.1.4.Referenceresolutiondependingondetermination . . . . . . 102
6.2.Actionreferenceresolution . . . . . . . . . . . . . . . . . . . . . 105
6.2.1.Actionreferenceandverbalsemantics . . . . . . . . . . . . 105
6.2.2.Analyzingtheutterance“putthatthere” . . . . . . . . . . . 108
6.3.Anaphoraandcoreferenceprocessing . . . . . . . . . . . . . . . 109
6.4.Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
Chapter7.DialogueActsRecognition . . . . . . . . . . . . . . . . . . 113
7.1.Natureofdialogueacts . . . . . . . . . . . . . . . . . . . . . . . . 114
7.1.1.Definitionsandphenomena . . . . . . . . . . . . . . . . . . 114
7.1.2.Theissuewithindirectacts . . . . . . . . . . . . . . . . . . 116
7.1.3.Theissuewithcompositeacts . . . . . . . . . . . . . . . . . 118
7.2.Identificationandprocessingofdialogueacts . . . . . . . . . . . 119
7.2.1.Actidentificationandclassification . . . . . . . . . . . . . . 119
7.2.2.Indirectandcompositeacts . . . . . . . . . . . . . . . . . . 121
7.3.Multimodaldialogueactprocessing . . . . . . . . . . . . . . . . 122
7.4.Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
viii Man–MachineDialogue
PART 3. SYSTEM BEHAVIOR AND EVALUATION . . . . . . . . . . . 125
Chapter8.AFewDialogueStrategies . . . . . . . . . . . . . . . . . . 127
8.1.Naturalandcooperativeaspectsofdialoguemanagement . . . . 128
8.1.1.Commongoalandcooperation . . . . . . . . . . . . . . . . 128
8.1.2.Speakingturnsandinteractiveaspects . . . . . . . . . . . . 130
8.1.3.Interpretationandinferences . . . . . . . . . . . . . . . . . . 132
8.1.4.Dialogue,argumentationandcoherence . . . . . . . . . . . 133
8.1.5.Choosingananswer. . . . . . . . . . . . . . . . . . . . . . . 135
8.2.Technicalaspectsofdialoguemanagement . . . . . . . . . . . . 136
8.2.1.Dialoguemanagementandcontrol . . . . . . . . . . . . . . 136
8.2.2.Dialoguehistorymodeling . . . . . . . . . . . . . . . . . . . 138
8.2.3.Dialoguemanagementandmultimodalitymanagement. . . 143
8.2.4.Canadialoguesystemlie? . . . . . . . . . . . . . . . . . . . 145
8.3.Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
Chapter9.MultimodalOutputManagement . . . . . . . . . . . . . . 149
9.1.Outputmanagementmethodology . . . . . . . . . . . . . . . . . 151
9.1.1.Generalprinciplesofoutputmultimodality . . . . . . . . . 151
9.1.2.Humanfactorsformultimediapresentation . . . . . . . . . 153
9.2.Multimediapresentationpragmatics . . . . . . . . . . . . . . . . 156
9.2.1.Illocutionaryforcesandvalues . . . . . . . . . . . . . . . . 156
9.2.2.Perlocutionaryforcesandvalues . . . . . . . . . . . . . . . 157
9.3.Processes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
9.3.1.Allocationoftheinformationover
communicationchannels . . . . . . . . . . . . . . . . . . . . 159
9.3.2.Redundancymanagementandmultimodalfission . . . . . . 161
9.3.3.Generationofreferringexpressions . . . . . . . . . . . . . . 162
9.3.4.Valorizingpartoftheinformationandtext
tospeechsynthesis. . . . . . . . . . . . . . . . . . . . . . . . 163
9.4.Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
Chapter10.MultimodalDialogueSystemAssessment . . . . . . . . 167
10.1.Dialoguesystemassessmentfeasibility . . . . . . . . . . . . . . 168
10.1.1.Afewassessmentexperiments . . . . . . . . . . . . . . . . 170
10.1.2.Man–machineinterfacemethodologies . . . . . . . . . . . 172
10.1.3.Oraldialoguemethodologies . . . . . . . . . . . . . . . . . 174
TableofContents ix
10.1.4.Multimodaldialoguemethodologies. . . . . . . . . . . . . 175
10.2.Multimodalsystemassessmentchallenges . . . . . . . . . . . . 176
10.2.1.Globalassessmentorsegmentedassessment? . . . . . . . 176
10.2.2.Shouldamultimodalcorpusbemanaged? . . . . . . . . . 178
10.2.3.Canwecompareseveralmultimodalsystems? . . . . . . . 179
10.3.Methodologicalelements . . . . . . . . . . . . . . . . . . . . . . 180
10.3.1.Userexpertiseandsystemcomplexity . . . . . . . . . . . . 181
10.3.2.Questionnairesforusers . . . . . . . . . . . . . . . . . . . 183
10.3.3.ExtendingDQRandDCRtomultimodaldialogue . . . . . 185
10.3.4.Towardsotherassessmentmethods . . . . . . . . . . . . . 189
10.4.Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
Preface
The preparation of this book was carried out while preparing an
accreditation to supervise research. This is a synthesis covering the past 10
years of research, since my doctorate [LAN 04], in the field of man–machine
dialogue. The goal here is to outline the theories, methods, techniques and
challenges involved in the design of computer programs that are able to
understand and produce speech. This synthesis covers the presentation of
important works in the field as well as a more personal approach, visible
through the choice of the themes explored, for example. How can a machine
talk, understand what is said and carry out a conversation close to natural
conversation between two human beings? What are the design stages of a
man–machine dialogue system? What are the understanding, thinking and
interaction abilities expected from such systems? How should they be
implemented? How can we get closer to the realistic and fluid aspect of
humandialogue?Canadialoguesystemlie?
These questions are at the origin of my path, which oscillated between
linguistics and computer science, between pure research and development,
between public and private research laboratories: INRIA, then Thales and
currently the CNRS. These are also questions that second-year Masters
students asked me during the man–machine dialogue class that I held at
University Paris Diderot for a few years. Thus, this book draws inspiration in
part from the class preparation and aims to be accessible to a public with
linguistic and automatic language processing notions, and not necessarily
withknowledgeoftheman–machinedialoguedomain.
xii Man–MachineDialogue
The goal here is to explain the main issues created by each stage of the
design of a man–machine dialogue system, and to showa fewtheoretical and
technicalpathsusedtodealwiththeseissues.Thepresentationwillnotcover
all the wealth of existing works, but beyond that, it will aim to provide the
readersaglimpseofthefield,whichmightmakethemwanttoknowmore.
The goal here is to show that today there still is a French school of
man–machine dialogue, which has been especially active in the past few
years, even if it was at times a bit slower and at times it appeared that the
man–machine dialogue was an aporia. The French school is characterized by
its multidisciplinary approach, its involvement in different fields, such as
system development (university prototypes, general public systems, as well
as – and we tend to forget them since they are confidential – military
systems), implementation of assessment methods and campaigns, and
software architecture design. There is a French school for multimodal
dialogue, for ergonomics, for embodied conversational agents, and even for
theapplicationofmachinelearningtechniquestotheman–machinedialogue.
Not all the links between these specialties are completely finalized, but the
generaldynamicsareundeniableandencouraging.
Asusualinresearchwork,whatispresentedinthisbookisindebtedtothe
encouragement, advice and more generally speaking the sharing of an
efficient and enjoyable work environment. For their institutional as well as
scientific and human encouragement, I would like to thank Francis Corblin,
Catherine Fuchs, Valérie Issarny, Jean-Marie Pierrel, Laurent Romary,
Jean-PaulSansonnet,CatherineSchnedecker,JacquesSiroux,MariëtTheune,
Bernard Victorri and Anne Vilnat. For the incredibly enriching Ozone
experiment during my postdoctorate fellowship at INRIA, I would
particularly like to thank Christophe Cérisara, Yves Laprie and especially
Alexandre Denis on whom I was able to rely to implement a memorable
demonstrator. For the equally memorable experiment of Thales R&T, I
would like to thank, more specifically, Claire Fraboulet-Laudy, Bénédicte
Goujon, Olivier Grisvard, Jérôme Lard and Célestin Sedogbo. For the
wonderful workplace that is the Lattice laboratory, a Joint Research Unit of
the CNRS, I would like to thank, without repeating those whom I have
already mentioned, Michel Charolles for our very enriching exchanges on
reference, Shirley Carter-Thomas and Sophie Prévost for the information
structure, Thierry Poibeau and Isabelle Tellier for natural language
processing, my successive colleagues Sylvain, Laure, Frédérique, as well as