Computational Paralinguistics: Emotion, Affect and Personality in Speech and Language Processing

345 Pages·2013·5.22 MB·English

by  Björn W. Schuller

Björn Schuller | Anton BAtliner
Computational
paralinguistiCs
Emotion, affECt and pErsonality in
spEECh and languagE proCEssing

COMPUTATIONAL
PARALINGUISTICS

COMPUTATIONAL
PARALINGUISTICS
EMOTION, AFFECT AND PERSONALITY IN
SPEECH AND LANGUAGE PROCESSING
Bjo¨rnW.Schuller
TechnischeUniversita¨tMu¨nchen,Germany
AntonM.Batliner
Friedrich-Alexander-Universita¨tErlangen-Nu¨rnberg,Germany
Thiseditionfirstpublished2014
(cid:2)C 2014JohnWiley&Sons,Ltd
Registeredoffice
JohnWiley&SonsLtd,TheAtrium,SouthernGate,Chichester,WestSussex,PO198SQ,UnitedKingdom
Fordetailsofourglobaleditorialoffices,forcustomerservicesandforinformationabouthowtoapplyfor
permissiontoreusethecopyrightmaterialinthisbookpleaseseeourwebsiteatwww.wiley.com.
TherightoftheauthortobeidentifiedastheauthorofthisworkhasbeenassertedinaccordancewiththeCopyright,
DesignsandPatentsAct1988.
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmitted,inany
formorbyanymeans,electronic,mechanical,photocopying,recordingorotherwise,exceptaspermittedbytheUK
Copyright,DesignsandPatentsAct1988,withoutthepriorpermissionofthepublisher.
Wileyalsopublishesitsbooksinavarietyofelectronicformats.Somecontentthatappearsinprintmaynotbe
availableinelectronicbooks.
Designationsusedbycompaniestodistinguishtheirproductsareoftenclaimedastrademarks.Allbrandnamesand
productnamesusedinthisbookaretradenames,servicemarks,trademarksorregisteredtrademarksoftheir
respectiveowners.Thepublisherisnotassociatedwithanyproductorvendormentionedinthisbook.
LimitofLiability/DisclaimerofWarranty:Whilethepublisherandauthorhaveusedtheirbesteffortsinpreparing
thisbook,theymakenorepresentationsorwarrantieswithrespecttotheaccuracyorcompletenessofthecontentsof
thisbookandspecificallydisclaimanyimpliedwarrantiesofmerchantabilityorfitnessforaparticularpurpose.Itis
soldontheunderstandingthatthepublisherisnotengagedinrenderingprofessionalservicesandneitherthe
publishernortheauthorshallbeliablefordamagesarisingherefrom.Ifprofessionaladviceorotherexpert
assistanceisrequired,theservicesofacompetentprofessionalshouldbesought.
LibraryofCongressCataloging-in-PublicationData
Schuller,Bjo¨rn.
Computationalparalinguistics:emotion,affectandpersonalityinspeechandlanguage
processing/Bjo¨rnSchuller,AntonBatliner.–FirstEdition.
pagescm
Includesbibliographicalreferencesandindex.
ISBN978-1-119-97136-8(hardback)
1.Psycholinguistics–Dataprocessing. 2.Linguisticmodels–Dataprocessing. 3.Paralinguistics.
4.Languageandemotions. 5.Speechprocessingsystems. 6.Human-computerinteraction.
7.Emotive(Linguistics) 8.Computationallinguistics. I.Batliner,Anton. II.Title.
P37.5.D37S382014
401(cid:3).90285–dc23
2013019325
AcataloguerecordforthisbookisavailablefromtheBritishLibrary.
Typesetin10/12ptTimesbyAptaraInc.,NewDelhi,India
1 2014
ToDagmarandGisela

Contents
Preface xiii
Acknowledgements xv
ListofAbbreviations xvii
Part I Foundations
1 Introduction 3
1.1 WhatisComputationalParalinguistics?AFirstApproximation 3
1.2 HistoryandSubjectArea 7
1.3 FormversusFunction 10
1.4 FurtherAspects 12
1.4.1 TheSynthesisofEmotionandPersonality 12
1.4.2 Multimodality:AnalysisandGeneration 13
1.4.3 Applications,UsabilityandEthics 15
1.5 SummaryandStructureoftheBook 17
References 18
2 Taxonomies 21
2.1 TraitsversusStates 21
2.2 ActedversusSpontaneous 25
2.3 ComplexversusSimple 30
2.4 MeasuredversusAssessed 31
2.5 CategoricalversusContinuous 33
2.6 FeltversusPerceived 35
2.7 IntentionalversusInstinctual 37
2.8 ConsistentversusDiscrepant 38
2.9 PrivateversusSocial 39
2.10 PrototypicalversusPeripheral 40
2.11 UniversalversusCulture-Specific 41
2.12 UnimodalversusMultimodal 43
2.13 AllTheseTaxonomies–SoWhat? 44
2.13.1 EmotionData:TheFAUAEC 45
2.13.2 Non-nativeData:TheC-AuDiTcorpus 47
References 48
viii Contents
3 AspectsofModelling 53
3.1 TheoriesandModelsofPersonality 53
3.2 TheoriesandModelsofEmotionandAffect 55
3.3 TypeandSegmentationofUnits 58
3.4 TypicalversusAtypicalSpeech 60
3.5 Context 61
3.6 LabversusLife,orThroughtheLookingGlass 62
3.7 SheepandGoats,orSingleInstanceDecisionversusCumulativeEvidence
andOverallPerformance 64
3.8 TheFewandtheMany,orHowtoAnalyseaHamburger 65
3.9 Reifications,andWhatYouareLookingforisWhatYouGet 67
3.10 MagicalNumbersversusSoundReasoning 68
References 74
4 FormalAspects 79
4.1 TheLinguisticCodeandBeyond 79
4.2 TheNon-DistinctiveUseofPhoneticElements 81
4.2.1 SegmentalLevel:TheCaseof/r/Variants 81
4.2.2 Supra-segmentalLevel:TheCaseofPitchandFundamental
Frequency–andofOtherProsodicParameters 82
4.2.3 InBetween:TheCaseofOtherVoiceQualities,Especially
Laryngealisation 86
4.3 TheNon-DistinctiveUseofLinguisticsElements 91
4.3.1 WordsandWordClasses 91
4.3.2 PhraseLevel:TheCaseofFillerPhrasesandHedges 94
4.4 Disfluencies 96
4.5 Non-Verbal,VocalEvents 98
4.6 CommonTraitsofFormalAspects 100
References 101
5 FunctionalAspects 107
5.1 BiologicalTraitPrimitives 109
5.1.1 SpeakerCharacteristics 111
5.2 CulturalTraitPrimitives 112
5.2.1 SpeechCharacteristics 114
5.3 Personality 115
5.4 EmotionandAffect 119
5.5 SubjectivityandSentimentAnalysis 123
5.6 DeviantSpeech 124
5.6.1 PathologicalSpeech 125
5.6.2 TemporarilyDeviantSpeech 129
5.6.3 Non-nativeSpeech 130
5.7 SocialSignals 131
5.8 DiscrepantCommunication 135
5.8.1 IndirectSpeech,Irony,andSarcasm 136
5.8.2 DeceptiveSpeech 138
5.8.3 Off-Talk 139