Human Behavior Analysis: Sensing and Understanding

277 Pages·2020·8.155 MB·English

by  Zhiwen Yu, Zhu Wang

Zhiwen Yu
Zhu Wang
Human Behavior
Analysis:
Sensing and
Understanding
Human Behavior Analysis: Sensing
and Understanding
(cid:129)
Zhiwen Yu Zhu Wang
Human Behavior Analysis:
Sensing and Understanding
ZhiwenYu ZhuWang
SchoolofComputerScience SchoolofComputerScience
NorthwesternPolytechnicalUniversity NorthwesternPolytechnicalUniversity
Xi’an,China Xi’an,China
ISBN978-981-15-2108-9 ISBN978-981-15-2109-6 (eBook)
https://doi.org/10.1007/978-981-15-2109-6
©SpringerNatureSingaporePteLtd.2020
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartofthe
materialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors, and the editorsare safeto assume that the adviceand informationin this
bookarebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSingaporePteLtd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore
Preface
Inrecentyears,humanbehaviorsensingandunderstandingattractsalotofinterests
due to various societal needs, including security, natural interfaces, gaming, affec-
tivecomputing,andassistedliving.However,accuratedetectionandrecognitionof
humanbehaviorisstillabigchallengethatattractsalotofresearchefforts.
Traditionally,toidentifyhumanbehaviors,wefirstneedtocontinuouslycollect
the sensory data from physical sensing devices (e.g., camera, GPS, and RFID),
which can be either worn by humans, attached on objects, or deployed inenviron-
ments. Afterwards, based on recognition algorithms or classification models, the
behaviortypescanbeidentifiedsoastofacilitateupperlayerapplications.Although
such traditional behavior identification approaches perform well and are widely
adopted, most of them are intrusive and require specific sensing devices, raising
issuessuchasprivacyanddeploymentcost.
Inthismonograph,weaimtoprovideanoverviewofrecentresearchprogresson
noninvasive human behavior sensing and understanding. Specifically, this mono-
graphdiffersfromexistingliteratureinthefollowingaspects.Ontheonehand,we
mainly focus on human behavior understanding approaches that are based on
noninvasive sensing technologies, including both sensor-based and device-free
approaches. On the other hand, while most existing studies are about individual
behaviors,we willsystematicallyelaboratehowtounderstand human behaviorsof
differentgranularities,includingnotonlyindividual-levelbehaviorsbutalsogroup-
levelandcommunity-levelbehaviors.
Thebookincludesfourparts.InPartI(Chaps.1and2),weintroduceandanalyze
the design, implementation, and development of a typical human behavior sensing
and understanding system and then give the main steps of such a system. Part II
(Chaps. 3 and 4) mainly focuses on two noninvasive (i.e., sensor-based and
device-free) behavior sensing approaches. In Part III (Chaps. 5–7), we elaborate
our studies on the understanding of different granularity human behaviors, from
individualleveltogrouplevelandcommunitylevel.Finally,inPartIV(Chap.8),we
discusstheopenissuesandpossiblesolutionsinvolvedinhumanbehaviorsensing
andunderstanding,followedbyaconclusiontothewholemonograph.Specifically,
v
vi Preface
some of the contents in this monograph might be of particular interest to readers,
including noninvasive human behavior sensing approaches (i.e., sensor-based and
device-free), aswell astheunderstandingofdifferent granularityhumanbehaviors
(i.e.,individuallevel,grouplevel,andcommunitylevel).
WewouldliketothankProf.DaqingZhangattheSoftwareEngineeringInstitute
ofPekingUniversity,Beijing,China;Prof.LimingChenattheSchoolofComputer
Science and Informatics of De Montfort University, Leicester, UK; Prof. Xingshe
ZhouattheSchoolofComputerScienceofNorthwesternPolytechnicalUniversity,
Xi’an,China;andProf.BinGuoattheSchoolofComputerScienceofNorthwestern
PolytechnicalUniversity,Xi’an,China.Wewouldliketothankallofthemembers
of Ubiquitous Computing group of Northwestern Polytechnical University, China,
fortheirvaluablediscussions,insights,andhelpfulcomments.Wewouldalsoliketo
thank the staff at Springer, Ms. Celine Chang and Ms. Jane Li, for their kind help
throughoutthepublicationandpreparationprocessesofthemonograph.
Xi’an,China ZhiwenYu
Xi’an,China ZhuWang
Contents
1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 FromVision-BasedtoSensor-BasedandDevice-FreeBehavior
Sensing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 Vision-BasedHumanBehaviorSensingandRecognition. . . . 2
1.1.2 Sensor-BasedHumanBehaviorSensingandRecognition.... 3
1.1.3 Device-FreeHumanBehaviorSensingandRecognition. . . 4
1.2 FromIndividualtoGroupandCommunityBehaviorRecognition. . . 5
1.3 FromPattern-BasedtoModel-BasedBehaviorRecognition. . . . . . 7
1.3.1 Pattern-BasedBehaviorRecognition. . . . . . . . . . . . . . . . . 7
1.3.2 Model-BasedBehaviorRecognition. . . . . . . . . . . . . . . . . 8
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2 MainStepsofHumanBehaviorSensingandUnderstanding. . . . . . 13
2.1 SensoryDataCollection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2 DataPreprocessing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3 FeatureExtraction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.4 HumanBehaviorModelingandClassification. . . . . . . . . . . . . . . 16
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3 Sensor-BasedBehaviorRecognition. . . . . . . . . . . . . . . . . . . . . . . . . 17
3.1 Sensor-BasedBehaviorRecognitionEvolution. . . . . . . . . . . . . . . 17
3.2 BehaviorRecognitionBasedonMobileDevices. . . . . . . . . . . . . . 18
3.2.1 BehaviorSensingandUnderstandingScales. . . . . . . . . . . 19
3.2.2 BehaviorSensingandUnderstandingParadigms. . . . . . . . 20
3.3 Energy-EfficientBehaviorRecognitionUsingUbiquitousSensors. . . 21
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4 Device-FreeBehaviorRecognition. . . . .. . . . . . .. . . . . . .. . . . . .. . 27
4.1 TheBasicConceptofDevice-FreeBehaviorSensingand
Recognition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.1.1 GeneralMethodology. . . . . . . . . . . . . . . . . . . . . . . . . . . 28
vii
viii Contents
4.1.2 TypicalApplications. . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.2 Wi-FiCSI-BasedBehaviorSensingandRecognition. . . . . . . . . . 29
4.3 Acoustic-BasedBehaviorSensingandRecognition. . . . . . . . . . . . 32
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5 IndividualBehaviorRecognition. . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.1 HumanMobilityPredictionbyExploringHistoryTrajectories. . . . 37
5.1.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.1.2 RelatedWork. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.1.3 SerendipitousSocialInteractionsSupportingSystem. . . . . 40
5.1.4 Application. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.1.5 PerformanceEvaluation. . . . . . . . . . . . . . . . . . . . . . . . . . 46
5.2 DisorientationDetectionbyMiningGPSTrajectories. . . . . . . . . . 48
5.2.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
5.2.2 RelatedWork. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
5.2.3 DisorientationDetectionProblemFormulation. . . . . . . . . . 53
5.2.4 iBDD:Isolation-BasedDisorientationDetection. . . . . . . . 56
5.2.5 DisorientationTrajectoryDetectionAlgorithm. . . . . . . . . . 61
5.2.6 PerformanceEvaluation. . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.3 HumanComputerOperationRecognitionBasedonSmartphone. . . . 70
5.3.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
5.3.2 RelatedWork. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
5.3.3 SystemOverview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.3.4 KeystrokeIdentification. . . . . . . . . . . . . . . . . . . . . . . . . . 74
5.3.5 WordCorrection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
5.3.6 Human-ComputerOperationRecognition. . . . . . . . . . . . . 80
5.3.7 PerformanceEvaluation. . . . . . . . . . . . . . . . . . . . . . . . . . 82
5.4 SwimmerLocalizationBasedonSmartphone. . . . . . . . . . . . . . . . 85
5.4.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.4.2 RelatedWork. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
5.4.3 SystemArchitecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
5.4.4 SwimmingBehaviorRecognition. . . . . . . . . . . . . . . . . . . 88
5.4.5 SwimmerLocating. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
5.4.6 PerformanceEvaluation. . . . . . . . . . . . . . . . . . . . . . . . . . 95
5.5 HumanIdentityRecognitionBasedonWi-FiSignals. . . . . . . . . . 98
5.5.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
5.5.2 RelatedWork. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
5.5.3 ProblemAnalysisandSystemFramework. . . . . . . . . . . . . 101
5.5.4 DetailedDesignofHumanIdentification. . . . . . . . . . . . . . 103
5.5.5 PerformanceEvaluation. . . . . . . . . . . . . . . . . . . . . . . . . . 107
5.6 C-FMCW-BasedContactlessRespirationDetectionUsing
AcousticSignals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
5.6.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
5.6.2 RelatedWork. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.6.3 C-FMCW:AHigh-ResolutionDistanceEstimation
Method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
Contents ix
5.6.4 ContactlessRespirationDetectionUsingC-FMCW
withCommodityAcousticDevices. . . . . . . . . . . . . . . . . . 119
5.6.5 PerformanceEvaluation. . . . . . . . . . . . . . . . . . . . . . . . . . 124
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
6 GroupBehaviorRecognition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
6.1 RecognitionofGroupMobilityLevelandGroupStructure
withMobileDevices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
6.1.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
6.1.2 RelatedWork. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
6.1.3 SystemOverview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
6.1.4 GroupMobilityClassification. . . . . . . . . . . . . . . . . . . . . . 144
6.1.5 GroupStructureRecognition. . . . . . . . . . . . . . . . . . . . . . 145
6.1.6 PerformanceEvaluation. . . . . . . . . . . . . . . . . . . . . . . . . . 152
6.2 RecognitionofGroupSemanticInteractions. . . . . . . . . . . . . . . . . 154
6.2.1 SocialSemantics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
6.2.2 GatheringMultimodalMeetingContent. . . . . . . . . . . . . . 156
6.2.3 RecognizingtheSocialSemantics. . . . . . . . . . . . . . . . . . . 156
6.2.4 MiningSocialSemantics. . . . . . . . . . . . . . . . . . . . . . . . . 158
6.2.5 Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
6.3 RecognitionofGroupInteractionPatterns. . . . . . . . . . . . . . . . . . 165
6.3.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
6.3.2 RelatedWork. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
6.3.3 HumanSemanticInteraction. . . . . . . . . . . . . . . . . . . . . . . 168
6.3.4 SystemArchitecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
6.3.5 CollaborativeInteractionCapture. . . . . . . . . . . . . . . . . . . 170
6.3.6 MultimodalInteractionRecognition. . . . . . . . . . . . . . . . . 171
6.3.7 PerformanceEvaluation. . . . . . . . . . . . . . . . . . . . . . . . . . 173
6.4 GroupActivityOrganizationandSuggestionwithMobileCrowd
Sensing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
6.4.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
6.4.2 RelatedWork. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
6.4.3 GroupActivityModeling. . . . . . . . . . . . . . . . . . . . . . . . . 177
6.4.4 MobiGroupArchitecture. . . . . . . . . . . . . . . . . . . . . . . . . 180
6.4.5 PlannedGroupActivityPreparation. . . . . . . . . . . . . . . . . 182
6.4.6 RunningActivityRecognitionandSuggestion. . . . . . . . . . 188
6.4.7 PerformanceEvaluation. . . . . . . . . . . . . . . . . . . . . . . . . . 192
6.5 PredictingActivityAttendanceinMobileSocialNetworks. . . . . . 194
6.5.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
6.5.2 RelatedWork. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
6.5.3 ProblemStatementandSystemOverview. . . . . . . . . . . . . 197
6.5.4 FeatureModeling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
6.5.5 PerformanceEvaluation. . . . . . . . . . . . . . . . . . . . . . . . . . 207
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
x Contents
7 CommunityBehaviorUnderstanding. . . . . . . . . . . . . . . . . . . . . . . . 219
7.1 DiscoveringCommunitiesinMobileSocialNetworks. . . . . . . . . . 219
7.1.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
7.1.2 RelatedWork. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
7.1.3 ProblemStatement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
7.1.4 MultimodeMulti-AttributeEdge-CentricCo-clustering
Framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
7.1.5 EmpiricalStudyBasedonFoursquare. . . . . . . . . . . . . . . . 232
7.1.6 PerformanceEvaluation. . . . . . . . . . . . . . . . . . . . . . . . . . 234
7.2 UnderstandingSocialRelationshipEvolutionbyUsing
Real-WorldSensingData. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
7.2.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
7.2.2 RelatedWork. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
7.2.3 FriendshipPrediction. . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
7.2.4 SocialRelationshipEvolution. . . . . . . . . . . . . . . . . . . . . . 241
7.2.5 PerformanceEvaluation. . . . . . . . . . . . . . . . . . . . . . . . . . 242
7.3 InterlinkingOff-LineandOnlineCommunities. . . . . . . . . . . . . . . 245
7.3.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
7.3.2 RelatedWork. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
7.3.3 AnOverviewofHSN. . . . . . . . . . . . . . . . . . . . . . . . . . . 248
7.3.4 DetailedDesignofHSN. . . . . . . . . . . . . . . . . . . . .. . . . . 251
7.3.5 PerformanceEvaluation. . . . . . . . . . . . . . . . . . . . . . . . . . 254
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
8 OpenIssuesandEmergingTrends. . . . . . . . . . . . . . . . . . . . . . . . . . 261
8.1 ResearchChallenges. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
8.1.1 ChallengesfromHumanBehaviorItself. . . . . . . . . . . . . . 261
8.1.2 ChallengesfromtheData. . . . . . . . . . . . . . . . . . . . . . . . . 262
8.1.3 ChallengesfromModelingandEvaluation. . . . . . . . . . . . 262
8.1.4 TenMostImportantProblems. . . . . . . . . . . . . . . . . . . . . 263
8.2 EmergingTrendsandDirections. . . . . . . . . . . . . . . . . . . . . . . . . 266
8.2.1 ComplexBehaviorRecognition. . . . . . . . . . . . . . . . . . . . 266
8.2.2 MultilevelBehaviorModelingforScalability
andReusability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
8.2.3 AbnormalBehaviorRecognition. . . . . . . . . . . . . . . . . . . . 268
8.2.4 IntentorGoalRecognition. . . . . . . . . . . . . . . . . . . . . . . . 268
8.2.5 SensorDataReuseandRepurposing. . . . . . . . . . . . . . . . . 269
8.3 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270