Advanced Man-Machine Interaction: Fundamentals and Implementation

471 Pages·2006·15.981 MB·English

by  Prof. Dr.-Ing. Karl-Friedrich Kraiss (auth.), Professor Dr.-Ing. Karl-Friedrich Kraiss (eds.)

Karl-FriedrichKraiss(Ed.)
AdvancedMan-MachineInteraction
Karl-Friedrich Kraiss (Ed.)
Advanced Man-Machine
Interaction
FundamentalsandImplementation
With280Figures
123
Editor
ProfessorDr.-Ing.Karl-FriedrichKraiss
RWTHAachenUniversity
ChairofTechnicalComputerScience
Ahornstraße55
52074Aachen
Germany
[email protected]
LibraryofCongressControlNumber:2005938888
ISBN-10 3-540-30618-8 SpringerBerlinHeidelbergNewYork
ISBN-13 978-3-540-30618-4 SpringerBerlinHeidelbergNewYork
Thisworkissubjecttocopyright.Allrightsarereserved,whetherthewholeorpartofthematerial
is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting,reproductiononmicrofilmorinanyotherway,andstorageindatabanks.Duplication
ofthispublicationorpartsthereofispermittedonlyundertheprovisionsoftheGermanCopyright
LawofSeptember9,1965,initscurrentversion,andpermissionforusemustalwaysbeobtainedfrom
Springer.ViolationsareliableforprosecutionundertheGermanCopyrightLaw.
SpringerisapartofSpringerScience+BusinessMedia
springer.com
©Springer-VerlagBerlinHeidelberg2006
PrintedinGermany
Theuseofgeneraldescriptivenames,registerednames,trademarks,etc.inthispublicationdoesnot
imply, even in the absence of a specific statement, thatsuch names are exempt from the relevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Typesetting:Digitaldatasuppliedbyeditor
CoverDesign:design&productionGmbH,Heidelberg
Production:LE-TEXJelonek,Schmidt&VöcklerGbR,Leipzig
Printedonacid-freepaper 7/3100/YL 543210
To:Eva,
Gregor,
Lucas,
Martin,and
Robert.
Preface
Thewayinwhichhumansinteractwithmachineshaschangeddramaticallyduring
thelastfifty years.Drivenbydevelopmentsin softwareandhardwaretechnologies
new appliances never thought of some years ago appear on the market. There are
apparentlyno limits to the functionalityof personalcomputersor mobile telecom-
municationappliances.Intelligenttrafficsystemspromiseformerlyunknownlevels
ofsafety.Cooperationbetweenfactoryworkersandsmartautomationisunderway
inindustrialproduction.Telepresenceandteleoperationsinspace,underwaterorin
microworldsismadepossible.Mobilerobotsprovideservicesondutyandathome.
As,unfortunately,humanevolutiondoesnotkeeppacewithtechnology,weface
problemsindealingwiththisbravenewworld.Manyindividualshaveexperienced
minorormajorcalamitieswhenusingcomputersforbusinessorinprivate.Theel-
derly and people with special needs often refrain altogether from capitalizing on
modern technology. Statistics indicate that in traffic systems at least two of three
accidentsarecausedbyhumanerror.
In this situation man-machineinterfacedesign is widely recognizedas a major
challenge. It represents a key technology,which promises excellent pay-offs.As a
majormarketingfactoritalsosecuresacompetitiveedgeovercontenders.
Thepurposeofthisbookistoprovidedeepinsightintonovelenablingtechnolo-
gies of man-machine interfaces. This includes multimodal interaction by mimics,
gesture, and speech, video based biometrics, interaction in virtual reality and with
robots,andtheprovisionofuserassistance.Alltheseareinnovativeandstilldynam-
icallydevelopingresearchareas.
The concept of this book is based on my lectures on man-machine systems
heldregularlyforstudentsinelectricalengineeringandcomputerscienceatRWTH
AachenUniversityandonresearchperformedduringthelastyearsatthischair.
Thecompilationofthematerialswouldnothavebeenpossiblewithoutcontribu-
tionsof numerouspeoplewho workedwith me overthe yearson varioustopicsof
man-machineinteraction.Severalchaptersareauthoredbyformerorcurrentdoctoral
students. Some chaptersbuild in parton earlier work of formercoworkersas, e.g.,
Suat Akyol, Pablo Alvarado, Ingo Elsen, Kirsti Grobel, Hermann Hienz, Thomas
Kru¨ger,DirkKrumbiegel,RolandSteffan,PeterWalter,andJochenWickel.There-
foretheirworkdeservesmentioningaswell.
To providea well balancedand complete view on the topic the volume further
contains two chapters by Professor Gerhard Rigoll from Technical University of
Munich and by Professor Ru¨diger Dillmann from Karlsruhe University. I’m very
muchindebtedtobothfortheircontributions.Ialsowishtoexpressmygratitudeto
EvaHestermann-BeyerleandtoMonikaLempefromSpringerVerlag:totheformer
for triggering the writing of this book, and to both for provided helpful support
duringitspreparation.
VIII Preface
I am deeply indebted to Jo¨rg Zieren for administrating a central repository for
the various manuscripts and for making sure that the guidelines were accurately
followed.AlsoLarsLibudadeservessincerethanksforundertakingthetediouswork
ofcomposingandtesting thebooksCD.Theassistance ofbothwasa fundamental
prerequisiteformeetingthegivendeadline.
FinallyspecialthanksgotomywifeCorneliaforunwaveringsupport,and,more
general,forsharinglifewithme.
Aachen,December2005 Karl-FriedrichKraiss
Table of Contents
1 Introduction.................................................. 1
2 Non-IntrusiveAcquisitionofHumanAction ....................... 7
2.1 HandGestureCommands ........................................ 7
2.1.1 ImageAcquisitionandInputData......................... 8
2.1.1.1 Vocabulary ..................................... 8
2.1.1.2 RecordingConditions ............................ 9
2.1.1.3 ImageRepresentation ............................ 10
2.1.1.4 Example ....................................... 11
2.1.2 FeatureExtraction...................................... 12
2.1.2.1 HandLocalization ............................... 14
2.1.2.2 RegionDescription .............................. 20
2.1.2.3 GeometricFeatures .............................. 22
2.1.2.4 Example ....................................... 27
2.1.3 FeatureClassification ................................... 29
2.1.3.1 ClassificationConcepts........................... 29
2.1.3.2 ClassificationAlgorithms ......................... 31
2.1.3.3 FeatureSelection ................................ 32
2.1.3.4 Rule-basedClassification ......................... 33
2.1.3.5 MaximumLikelihoodClassification ................ 34
2.1.3.6 ClassificationUsingHiddenMarkovModels......... 37
2.1.3.7 Example ....................................... 47
2.1.4 StaticGestureRecognitionApplication .................... 48
2.1.5 DynamicGestureRecognitionApplication ................. 50
2.1.6 Troubleshooting........................................ 56
2.2 FacialExpressionCommands..................................... 56
2.2.1 ImageAcquisition...................................... 59
2.2.2 ImagePreprocessing.................................... 60
2.2.2.1 FaceLocalization................................ 61
2.2.2.2 FaceTracking................................... 67
2.2.3 FeatureExtractionwithActiveAppearanceModels .......... 68
2.2.3.1 AppearanceModel............................... 73
2.2.3.2 AAMSearch.................................... 75
2.2.4 FeatureClassification ................................... 78
2.2.4.1 HeadPoseEstimation ............................ 78
2.2.4.2 DeterminationofLineofSight..................... 83
2.2.4.3 CircleHoughTransformation...................... 83
2.2.4.4 DeterminationofLipOutline ...................... 87
2.2.4.5 Lipmodeling ................................... 88
2.2.5 FacialFeatureRecognition–EyeLocalizationApplication.... 90
2.2.6 FacialFeatureRecognition–MouthLocalizationApplication . 90
X TableofContents
References ......................................................... 92
3 SignLanguageRecognition ..................................... 95
3.1 RecognitionofIsolatedSignsinReal-WorldScenarios................ 99
3.1.1 ImageAcquisitionandInputData......................... 101
3.1.2 ImagePreprocessing.................................... 102
3.1.2.1 BackgroundModeling............................ 102
3.1.3 FeatureExtraction...................................... 105
3.1.3.1 OverlapResolution .............................. 106
3.1.3.2 HandTracking .................................. 107
3.1.4 FeatureNormalization .................................. 109
3.1.5 FeatureClassification ................................... 110
3.1.6 TestandTrainingSamples ............................... 110
3.2 SignRecognitionUsingNonmanualFeatures........................ 111
3.2.1 NonmanualParameters.................................. 111
3.2.2 ExtractionofNonmanualFeatures ........................ 113
3.3 RecognitionofcontinuousSignLanguageusingSubunits ............. 116
3.3.1 SubunitModelsforSigns................................ 116
3.3.2 TranscriptionofSignLanguage........................... 118
3.3.2.1 Linguistics-orientatedTranscriptionofSignLanguage. 118
3.3.2.2 Visually-orientatedTranscriptionofSignLanguage ... 121
3.3.3 SequentialandParallelBreakdownofSigns ................ 122
3.3.4 ModificationofHMMstoParallelHiddenMarkovModels.... 122
3.3.4.1 ModelingSignLanguagebymeansofPaHMMs...... 124
3.3.5 Classification .......................................... 125
3.3.5.1 ClassificationofSingleSignsbyMeansofSubunits
andPaHMMs ................................... 125
3.3.5.2 ClassificationofContinuousSignLanguagebyMeans
ofSubunitsandPaHMMs......................... 126
3.3.5.3 StochasticLanguageModeling .................... 127
3.3.6 Training .............................................. 128
3.3.6.1 InitialTranscription.............................. 129
3.3.6.2 EstimationofModelParametersforSubunits ........ 131
3.3.6.3 ClassificationofSingleSigns...................... 132
3.3.7 ConcatenationofSubunitModelstoWordModelsforSigns. .. 133
3.3.8 EnlargementofVocabularySizebyNewSigns.............. 133
3.4 PerformanceEvaluation.......................................... 134
3.4.1 Video-basedIsolatedSignRecognition .................... 135
3.4.2 SubunitBasedRecognitionofSignsandContinuousSign
Language ............................................. 136
3.4.3 Discussion ............................................ 137
References ......................................................... 137
TableofContents XI
4 SpeechCommunicationandMultimodalInterfaces.................141
4.1 SpeechRecognition ............................................. 141
4.1.1 Fundamentalsof Hidden MarkovModel-basedSpeech
Recognition ........................................... 142
4.1.2 TrainingofSpeechRecognitionSystems ................... 144
4.1.3 RecognitionPhaseforHMM-basedASRSystems ........... 145
4.1.4 InformationTheoryInterpretationof Automatic Speech
Recognition ........................................... 147
4.1.5 SummaryoftheAutomaticSpeechRecognitionProcedure.... 149
4.1.6 SpeechRecognitionTechnology .......................... 150
4.1.7 ApplicationsofASRSystems ............................ 151
4.2 SpeechDialogs................................................. 153
4.2.1 Introduction ........................................... 153
4.2.2 InitiativeStrategies ..................................... 155
4.2.3 ModelsofDialog....................................... 156
4.2.3.1 FiniteStateModel ............................... 156
4.2.3.2 SlotFilling ..................................... 157
4.2.3.3 StochasticModel ................................ 158
4.2.3.4 GoalDirectedProcessing ......................... 159
4.2.3.5 RationalConversationalAgents.................... 160
4.2.4 DialogDesign ......................................... 162
4.2.5 ScriptingandTagging................................... 163
4.3 MultimodalInteraction .......................................... 164
4.3.1 In-andOutputChannels................................. 165
4.3.2 BasicsofMultimodalInteraction.......................... 166
4.3.2.1 Advantages..................................... 167
4.3.2.2 Taxonomy...................................... 167
4.3.3 MultimodalFusion ..................................... 168
4.3.3.1 IntegrationMethods.............................. 169
4.3.4 ErrorsinMultimodalSystems ............................ 172
4.3.4.1 ErrorClassification .............................. 172
4.3.4.2 UserSpecificErrors.............................. 172
4.3.4.3 SystemSpecificErrors ........................... 173
4.3.4.4 ErrorAvoidance................................. 173
4.3.4.5 ErrorResolution................................. 174
4.4 EmotionsfromSpeechandFacialExpressions....................... 175
4.4.1 Background ........................................... 175
4.4.1.1 ApplicationScenarios ............................ 175
4.4.1.2 Modalities...................................... 176
4.4.1.3 EmotionModel ................................. 176
4.4.1.4 EmotionalDatabases............................. 177
4.4.2 AcousticInformation ................................... 177
4.4.2.1 FeatureExtraction ............................... 178
4.4.2.2 FeatureSelection ................................ 179
4.4.2.3 ClassificationMethods ........................... 180
XII TableofContents
4.4.3 LinguisticInformation .................................. 180
4.4.3.1 N-Grams....................................... 181
4.4.3.2 Bag-of-Words................................... 182
4.4.3.3 PhraseSpotting ................................. 182
4.4.4 VisualInformation...................................... 183
4.4.4.1 Prerequisites.................................... 184
4.4.4.2 HolisticApproaches ............................. 184
4.4.4.3 AnalyticApproaches............................. 186
4.4.5 InformationFusion ..................................... 186
4.4.6 Discussion ............................................ 187
References ......................................................... 187
5 PersonRecognitionandTracking ................................191
5.1 FaceRecognition ............................................... 193
5.1.1 ChallengesinAutomaticFaceRecognition ................. 193
5.1.2 StructureofFaceRecognitionSystems..................... 196
5.1.3 CategorizationofFaceRecognitionAlgorithms ............. 200
5.1.4 GlobalFaceRecognitionusingEigenfaces.................. 201
5.1.5 Local Face Recognitionbased on Face Componentsor
TemplateMatching ..................................... 206
5.1.6 FaceDatabasesforDevelopmentandEvaluation ............ 212
5.1.7 Exercises ............................................. 214
5.1.7.1 ProvidedImagesandImageSequences.............. 214
5.1.7.2 FaceRecognitionUsingtheEigenfaceApproach ..... 215
5.1.7.3 FaceRecognitionUsingFaceComponents........... 221
5.2 Full-bodyPersonRecognition..................................... 224
5.2.1 StateoftheArtinFull-bodyPersonRecognition ............ 224
5.2.2 ColorFeaturesforPersonRecognition..................... 226
5.2.2.1 ColorHistograms................................ 226
5.2.2.2 ColorStructureDescriptor ........................ 227
5.2.3 TextureFeaturesforPersonRecognition ................... 228
5.2.3.1 OrientedGaussianDerivatives ..................... 228
5.2.3.2 HomogeneousTextureDescriptor .................. 229
5.2.4 ExperimentalResults ................................... 231
5.2.4.1 ExperimentalSetup .............................. 231
5.2.4.2 FeaturePerformance ............................. 231
5.3 Camera-basedPeopleTracking.................................... 232
5.3.1 Segmentation .......................................... 234
5.3.1.1 ForegroundSegmentationbyBackgroundSubtraction . 234
5.3.1.2 MorphologicalOperations ........................ 236
5.3.2 Tracking .............................................. 237
5.3.2.1 TrackingFollowingDetection ..................... 238
5.3.2.2 CombinedTrackingandDetection.................. 243
5.3.3 OcclusionHandling..................................... 248
5.3.3.1 OcclusionHandlingwithoutSeparation ............. 248