Analyzing Video Sequences of Multiple Humans: Tracking, Posture Estimation and Behavior Recognition

154 Pages·2002·11.7 MB·English

by  Jun Ohya

ANALYZING VIDEO SEQUENCES
OF MULTIPLE HUMANS
Tracking, Posture Estimation and
Behavior Recognition
THE KLUWER INTERNATIONAL SERIES IN
VIDEO COMPUTING
Series Editor
Mubarak Shah, Ph.D.
University ofCentral Florida
Orlando, USA
Video is a very powerful and rapidly changing medium. The
increasingavailabilityoflowcost, lowpower,highlyaccurate video
imagery has resulted in the rapid growth ofapplications using this
data. Video provides multiple temporal constraints, which make it
easier to analyze a complex, and coordinated series ofevents that
cannotbe understoodbyjustlooking atonly asingle image ora few
frames. The effective use ofvideo requires understanding ofvideo
processing, video analysis, video synthesis, video retrieval, video
compressionand otherrelated computingtechniques.
The Video Computing book series will provide a forum for the
dissemination of innovative research results for computer vision,
imageprocessing, databaseand computer graphicsresearchers, who
are interested in differentaspects ofvideo.
ANALYZING VIDEO SEQUENCES
OF MULTIPLE HUMANS
Tracking, Posture Estimation and
Behavior Recognition
Jun Ohya
Waseda University
Akira Utsumi
Advanced Telecommunications Research Institute
International
Junji Yamato
Nippon Telegraph & Telephone Corporation
SPRINGER SCIENCE+BUSINESS MEDIA, LLC
ISBN 978-1-4613-5346-1 ISBN 978-1-4615-1003-1 (eBook)
DOI 10.1007/978-1-4615-1003-1
Library of Congress Cataloging-in-Publication Data
A C.I.P. Catalogue record for this book is available
from the Library of Congress.
Copyright © 2002 by Springer Science+Business Media New York
Originally published by Kluwer Academic Publishers in 2002
Softcover reprint of the hardcover 1s t edition 2002
AU rights reserved. No part of this work may be reproduced, stored in a
retrieval system, or transmitted in any form or by any means, electronic,
mechanical, photocopying, microfilming, recording, or otherwise, without the
written permission from the Publisher, with the exception of any material
supplied specifically for the purpose of being entered and executed on a
computer system, for exclusive use by the purchaser of the work.
Printed on acid-free paper.
Contents
ListofFigures ix
ListofTables xiii
Preface xvii
ContributingAuthors xxi
1
Introduction 1
Jun Ohya
2
Trackingmultiplepersonsfrommultiplecameraimages 7
AkiraUtsumi
2.1 OVERVIEW 7
2.2 PREPARATION 9
2.2.1 MultipleObservationsWithMultipleCameras(Observation
Redundancy) 9
2.2.2 KalmanFiltering 12
2.3 FEATURESOFMULTIPLECAMERABASEDTRACKINGSyS-
TEM 15
2.4 ALGORITHMFORMULTIPLE-CAMERAHUMANTRACKING
SYSTEM 17
2.4.1 MotionTrackingOfMultipleTargets 17
2.4.2 FindingNewTargets 21
2.5 IMPLEMENTATION 22
2.5.1 SystemOverview 22
2.5.2 FeatureExtraction 23
2.5.3 FeatureMatchingatanObservationNode 26
2.6 EXPERIMENTS 26
2.7 DISCUSSIONANDCONCLUSIONS 33
Appendix: ImageSegmentationusingSequential-image-basedAdaptation 35
3
Postureestimation 43
Jun Ohya
vi ANALYZINGVIDEOSEQUENCESOFMULTIPLEHUMANS
3.1 Introduction 43
3.2 AHeuristicMethodforEstimatingPosturesin2D 46
3.2.1 Outline 46
3.2.2 Locatingsignificantpointsofthehumanbody 46
3.2.2.1 CenterofGravityoftheHumanBody 46
3.2.2.2 OrientationoftheUpperHalfoftheHumanBody 48
3.2.2.3 LocatingSignificantPoints 49
3.2.3 EstimatingMajorJointPositions 52
3.2.3.1 AGABasedEstimationAlgorithm 52
3.2.3.2 ElbowJointPosition 53
3.2.3.3 KneeJointPosition 53
3.2.4 ExperimentalResults andDiscussions 54
3.2.4.1 ExperimentalSystem 54
3.2.4.2 SignificantPointLocationResults 54
3.2.4.3 JointPositionEstimation 54
3.2.4.4 Real-timeDemonstration 57
3.2.5 Summary 57
3.3 AHeuristicMethodforEstimatingPosturesin3D 60
3.3.1 Outline 60
3.3.2 ImageProcessingforTopCamera 61
3.3.2.1 Rotation AngleoftheBody 61
3.3.2.2 SignificantPoints 63
3.3.3 EstimatingMajorJointPositions 65
3.3.4 3DReconstructionoftheSignificantPoints 66
3.3.5 ExperimentalResults andDiscussions 67
3.3.5.1 ExperimentalSystem 67
3.3.5.2 SignificantPointDetectionResults 67
3.3.6 Summary 69
3.4 ANon-heuristicMethodforEstimatingPostures in3D 70
3.4.1 Outline 70
3.4.2 LocatingSignificantPointsforEachImage 70
3.4.2.1 Contouranalysis 71
3.4.2.2 ThetrackingprocessusingKalmanfilterandsubtractionim-
ageprocessmg 73
3.4.3 3DReconstructionoftheSignificantPoints 77
3.4.3.1 Frontimage 77
3.4.3.2 Sideimage 77
3.4.3.3 Topimage 78
3.4.3.4 Estimating3Dcoordinates 79
3.4.4 ExperimentalResults 79
3.4.4.1 ExperimentalSystem 79
3.4.4.2 ExperimentalResults 80
3.4.5 Summary 81
3.5 ApplicationstoVirtualEnvironments 86
3.5.1 VirtualMetamorphosis 86
3.5.2 VirtualKabukiSystem 87
3.5.3 The"ShallWeDance?" system 92
3.6 DiscussionandConclusion 94
4
Recognizinghuman behaviorusingHiddenMarkovModels 99
Junji Yamato
4.1 Backgroundand overview 99
Contents vii
4.2 HiddenMarkovModels 102
4.2.1 Outline 102
4.2.2 Recognition 104
4.2.3 Learning 104
4.3 ApplyingHMMtotime-sequentialimages 105
4.4 Experiments 108
4.4.1 Experimentalconditionsandpre-processes 108
4.4.2 Experiment 1 111
4.4.2.1 Experimentalconditions 111
4.4.2.2 Results 111
4.4.3 Experiment2 111
4.4.3.1 Experimentalconditions 111
4.4.3.2 Results 112
4.5 Category-separatedvectorquantization 114
4.5.1 ProbleminVQ 114
4.5.2 Category-separatedVQ 114
4.5.3 Experiment 114
4.6 ApplyingImageDatabaseSearch 121
4.6.1 Processoverview 121
4.6.2 Experiment 1: EvaluationofDCT 121
4.6.3 Experiment2: Evaluationofprecision-recall 124
4.6.4 ExtractingamovingareausinganMC vector 126
4.7 Discussionand Conclusion 129
5
ConclusionandFutureWork 133
fun Ohya
Index 137
List ofFigures
1.1 Tracking,PostureEstimationandBehaviorRecognition 2
2.1 Two-dimensionalmotiontrackingusingaone-dimensional
observation 10
2.2 (A)Fully-independentobservationsand(B)fully-redundant
observations 11
2.3 Timestamps oftwo-cameraobservations 12
2.4 (A)Most-independentobservationsand(B)most-redundant
observations 12
2.5 Multiple-camerahuman tracking system 16
2.6 Observationmodel 17
2.7 Kalman-filtering-basedmatching 22
2.8 Systemdiagram 23
2.9 Segmentation withsequential-image-basedadaptation 24
2.10 Featureextraction 24
2.11 Non-synchronous observationwith multiple viewpoints 25
2.12 Trackingresultfor one person 27
2.13 Observationintervals 27
2.14 Trackingresultsfortwo persons 28
2.15 Inputimage sequences (two persons) 29
2.16 Trackingaccuracy fornon-linear motion (circularmotion) 31
2.17 Trackingresults (circularmotion: subjectA) 32
2.18 State Tracking Result (From the top: X position, Y po-
sition, human height, and detected motion state for one
person's motion) 33
2.19 Exampleofmotionstateextraction(Top: 'walking,' Mid-
dle: 'standing,' Bottom: 'sitting.' The horizontal line
denotes theextractedheadheight.) 34
2.A.1 HierarchicalAdaptation 35
2.A.2 CoarseSegmentation 36
2.A.3 PixelValueDistributions 37
x ANALYZINGVIDEO SEQUENCESOFMULTIPLEHUMANS
2.A.4 DetectingLow-levelInformation 39
2.A.5 Segmentationusing Intensity Information 40
2.A.6 Segmentationusing Intensity Information 41
3.1 Postureusedforthecalibration 47
3.2 Example ofathermal image (gray-levels representtem-
perature values) 47
3.3 Distance-transformed image with a silhouette contour
(gray-levels represent distance values: bright and dark
levels indicate large andsmallvalues, respectively) 48
3.4 gij image with PAD(the white line) 49
3.5 Findingthe temporary positionofthe head(thecirclein
this figure) 50
3.6 Local maximaofaskeletonimage(whitepixels) 50
3.7 Locatingtipoffoot 51
3.8 Contoursegmentsto locatethe tipofthehand 52
3.9 Located significant points (target images are generated
by using the 3-Dhumanbody model) 55
3.10 Experimental results and reproduced Kabuki characters
(left column: original thermal image; middle column:
locatedsignificantpoints; rightcolumn: reproduction in
a Kabuki avatar 56
3.11 Estimationresults ofanelbowjointposition 58
3.12 Estimationresults ofakneejointposition 59
3.13 Outlineofthe heuristic methodfor 3D postureestimation 61
3.14 PrincipalAxis (PA)andContourin the Top View 62
3.15 Posturefor InitialCalibrationinthe TopView 62
3.16 Candidate for the Topofthe Headin the Top View 63
3.17 Candidatefor the Tipofthe Footinthe TopView 64
3.18 Candidatefor HandTipin theTopView 66
3.19 OriginalTrinocularImages(upper-left: frontview,upper-
right: side view, lower-left: top view, lower-right: no image) 68
3.20 Silhouettes oftheOriginalTrinocularImages 68
3.21 Results ofLocating SignificantPointsin theTrinocularImages 69
3.22 Trinocularcamerasystem 71
3.23 Definition ofLt - scurve 72
3.24 Examples ofLt - scurveanalysis 74
3.25 Examples of Lt - s curve analysis ((a) original image,
(b)contourimage, (c) Lt- scurve,(d) k-curvature,(e)
~ - Scurve 75
ListofFigures xi
3.26 Estimating the rotation angle using the skeleton Image
inthe top view 78
3.27 Examples ofEstimatingPostures in 3D 82
3.28 Examples ofEstimatingPostures in30 83
3.29 Examples ofEstimatingPostures in3D 84
3.30 EvaluationofEstimatedPositions ofLeftHand 85
3.31 VirtualKabuki System 88
3.32 Windows forEstimatingFacialExpressions 89
3.33 OCTFeatureCalculation 89
3.34 Changesin OCTFeaturesandFacialComponents' Shapes 90
3.35 ReferenceFacialExpressionsCreatedby Artist 91
3.36 Examples ofFacialExpressionReproduction 91
3.37 Scenes ofthe VirtualKabuki System 92
3.38 The "ShallWe Dance?" system 93
3.39 Automatic FaceTracking 94
4.1 ConceptofHiddenMarkovModels 103
4.2 Processing flow 106
4.3 Meshfeature 107
4.4 Sampleimage sequence 108
4.5 Extractionofhuman region 109
4.6 Extractedhumanimages 109
4.7 Targettennisactions 110
4.8 Category separatedVQ 115
4.9 Feature vectorsequenceinfeature space 116
4.10 Codebookgeneratedfrom LBG Algorithm 117
4.11 Ergodic HMM and LRHMM 118
4.12 Recognition rate(1) 119
4.13 Recognition rate(2) 120
4.14 Processingflowofcontent-basedimagedatabaseretrieval
using HMM behaviorrecognition method 122
4.15 Extracting lowerfrequency portion ofDCTcoefficients 123
4.16 Recognition rates using DCTas afeature 124
4.17 Spottingbehaviorsby thresholding log-likelihood 125
4.18 Precision rateandrecallrate 126
4.19 Moving areaextractionusing MC 127
4.20 Movingareaextractionusing asimplethresholdofMe. 128
4.21 Extractedmoving areas usingacombinationofMC andOCT. 129
5.1 Tracking, PostureEstimationandBehaviorRecognition 134