Person Re-Identification And Intelligent Crowdsourcing With Applications In Public Safety

187 Pages·2017·15.48 MB·English

PERSON RE-IDENTIFICATION AND INTELLIGENT CROWDSOURCING
WITH APPLICATIONS IN PUBLIC SAFETY
A Dissertation
Submitted to the Faculty
of
Purdue University
by
Khalid Tahboub
In Partial Fulfillment of the
Requirements for the Degree
of
Doctor of Philosophy
August 2017
Purdue University
West Lafayette, Indiana
ii
THE PURDUE UNIVERSITY GRADUATE SCHOOL
STATEMENT OF DISSERTATION APPROVAL
Dr. Edward J. Delp, Chair
School of Electrical and Computer Engineering
Dr. Amy R. Reibman
School of Electrical and Computer Engineering
Dr. Mary L. Comer
School of Electrical and Computer Engineering
Dr. Zygmunt Pizlo
Department of Psychological Sciences
Approved by:
Dr. Venkataramanan Balakrishnan
Head of the School Graduate Program
iii
To my parents,
Khulusi & Majeda,
with my deepest gratitude
iv
ACKNOWLEDGMENTS
First and foremost, I would like to thank my doctoral advisor Professor Edward J.
Delp. This thesis would not have been possible without his encouragement, support,
guidance and criticism. I am grateful to him for the opportunity and privilege of
becoming a member of the Video and Image Processing Laboratory (VIPER). As
a Ph.D. student, Professor Delp has been my role mode. His technical depth and
breadth, ability to inspire, and encouragement to overcome obstacles have been truly
invaluable. During our numerous conference trips, the many drives to Chicago and
Cleveland, I have also enjoyed our nonacademic-related conversations from which I
learned a lot.
I would like to thank Professor Amy R. Reibman for her guidance and support.
I am very grateful for the opportunity to work with a world class researcher. Her
educational approach, attention to details, kindness and analytical skills have made
working with her an enjoyable learning experience.
I would like to thank Professor Mary Comer for her encouragement throughout
my graduate studies. Her support and guidance have been invaluable. I would also
like to thank Professor Zygmunt Pizlo for his insightful feedback and suggestions.
I would like to thank Dr. Neeraj Gadgil for being a great teammate for the
crowdsourcing and the content based video retrieval projects. I am thankful for his
friendship and support during my Ph.D. studies. I am also thankful to all of VIPER
lab members for their encouragement, support and friendships. In particularly, I
would like to thank Blanca Delgado and Dahjung Chung for their help in the vBOLO
project and for being great teammates. I also want to thank Javier Ribera for his
work on the crowdsourcing project and for being a great colleague. And thank you
to the rest of my former and current colleagues: Di Chen, Qingchaung (Cici) Chen,
Yuhao Chen, Jeehyun Choe, David Gu¨era Cobo, Shaobo Fang, Chichen Fu, Dr. Ye
v
He, David Ho, Joonsoo Kim, Soonam Lee, He Li, Chang Liu, Daniel Mas, Dr. Albert
Parra Pozo, Ruiting Shao, Yu Wang, Dr. Chang (Joy) Xu, Sri Kalyan Yarlagadda,
Jiaju Yue, Dr. Bin Zhao. I would also like to thank our lab visitors Thitiporn (Bee)
Pramoun and Kharittha (Poy) Thongkur.
I would like to thank the endowment of the Charles William Harrison Distinguished
Professorship at Purdue University for partially supporting the work on the content
based video retrieval project.
The Chicago LTE test results discussed in the introduction chapter were obtained
in cooperation with the Chicago Police Department, Motorola Solutions and the U.S.
Department of Homeland Security. We gratefully acknowledge their cooperation.
The images shown in the person re-identification chapter were obtained in co-
operation with the Greater Cleveland Regional Transit Authority. We gratefully
acknowledge their cooperation.
This work was partially supported by the Cisco University Research Program
Fund CG-#594368 through the Silicon Valley Community Foundation.
This dissertation was partially supported by the Visual Analytics for Command,
Control, and Interoperability Environments (VACCINE) center under award number
2009-ST-061-CI000. VACCINE is a U.S. Department of Homeland Security’s center of
excellence at Purdue University. I am grateful for their support of the crowdsourcing
project, vBOLO project and the Chicago LTE study.
vi
TABLE OF CONTENTS
Page
LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x
ABBREVIATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
ABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvi
1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Video Surveillance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Video Analytics in a Mobile or Networked Environment . . . . . . . . . 2
1.3 Person Re-Identification . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.4 Crowdsourcing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.5 Contributions Of The Thesis . . . . . . . . . . . . . . . . . . . . . . . . 10
1.6 Publications Resulting From Our Work . . . . . . . . . . . . . . . . . . 14
2 QUALITY-ADAPTIVE DEEP LEARNING
FOR PEDESTRIAN DETECTION . . . . . . . . . . . . . . . . . . . . . . 17
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2 Overview of Previous Work . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3 Pedestrian Detection Using Compressed Video Sequences . . . . . . . . 21
2.4 Proposed Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.4.1 Training RPN Using Compressed Video Sequences . . . . . . . . 23
2.4.2 Proposed Two-Stage CNN . . . . . . . . . . . . . . . . . . . . . 25
2.4.3 Quality Estimation . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.5 Experiments and Results . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3 ACCURACY PREDICTION
FOR PEDESTRIAN DETECTION . . . . . . . . . . . . . . . . . . . . . . . 31
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
vii
Page
3.2 Pedestrian Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.3 Proposed Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.3.1 Texture Descriptors . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.3.2 Random Forest . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.4 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4 PERSON RE-IDENTIFICATION . . . . . . . . . . . . . . . . . . . . . . . . 45
4.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.1.1 Ensemble Of Local Features . . . . . . . . . . . . . . . . . . . . 49
4.1.2 Local Maximal Occurrence Feature . . . . . . . . . . . . . . . . 50
4.1.3 Hierarchical Gaussian Descriptor . . . . . . . . . . . . . . . . . 51
4.1.4 Keep It Simple and Straightforward Metric Learning . . . . . . 52
4.1.5 Cross-View Quadratic Discriminant Analysis . . . . . . . . . . . 53
4.2 Data Collection and Public Datasets . . . . . . . . . . . . . . . . . . . 53
4.3 Proposed Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.3.1 An Ensemble Of Localized Patches . . . . . . . . . . . . . . . . 62
4.3.2 Person Re-Identification Using
A Patch-Based Appearance Model . . . . . . . . . . . . . . . . 66
4.3.3 Dual Linear Regression-Based Classification . . . . . . . . . . . 72
4.3.4 Metric Learning-Based Deformable Graph Matching . . . . . . . 74
4.4 Experimental Results and Discussion . . . . . . . . . . . . . . . . . . . 78
5 INTELLIGENT CROWDSOURCING . . . . . . . . . . . . . . . . . . . . . 88
5.1 Overview of Previous Work . . . . . . . . . . . . . . . . . . . . . . . . 88
5.2 System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
5.3 System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
5.3.1 Administrator Portal . . . . . . . . . . . . . . . . . . . . . . . . 94
5.4 Crowdsourcing for Forensic Analysis of Surveillance Video . . . . . . . 98
5.4.1 Web-Based Annotation Platform with Fast Object Tracking . 100
5.4.2 Human Detection and Re-Identification . . . . . . . . . . . . . 102
viii
Page
5.5 Crowdsourcing for Real-time Alerting . . . . . . . . . . . . . . . . . . 104
5.5.1 Crowd Flow Estimation . . . . . . . . . . . . . . . . . . . . . 104
5.5.2 Online Active Learning from Crowds . . . . . . . . . . . . . . 106
5.6 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
5.6.1 System Performance . . . . . . . . . . . . . . . . . . . . . . . 110
5.6.2 Trained Vs. Untrained Crowds . . . . . . . . . . . . . . . . . 111
5.6.3 Online Active Learning From Crowds . . . . . . . . . . . . . . 114
5.6.4 Crowdsourcing for Forensic Analysis of Surveillance Video . . 117
6 COMPRESSED DOMAIN VIDEO SIGNATURES . . . . . . . . . . . . . 121
6.1 Overview of Previous Work . . . . . . . . . . . . . . . . . . . . . . . 123
6.2 Proposed Video Signature . . . . . . . . . . . . . . . . . . . . . . . . 128
6.2.1 Motion Vectors in HEVC . . . . . . . . . . . . . . . . . . . . . 128
6.2.2 Projections on Random Matrices . . . . . . . . . . . . . . . . 129
6.2.3 Signature Generation . . . . . . . . . . . . . . . . . . . . . . . 131
6.2.4 Signature Matching . . . . . . . . . . . . . . . . . . . . . . . . 134
6.2.5 Video Retrieval From Mobile Devices:
How Much Content Is Enough? . . . . . . . . . . . . . . . . . 137
6.3 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
6.3.1 Signatures Matching . . . . . . . . . . . . . . . . . . . . . . . 139
6.3.2 Content Quantification . . . . . . . . . . . . . . . . . . . . . . 141
6.3.3 Video Retrieval Performance . . . . . . . . . . . . . . . . . . . 144
6.4 Conclusion and Future Work . . . . . . . . . . . . . . . . . . . . . . . 144
7 SUMMARY AND FUTURE WORK . . . . . . . . . . . . . . . . . . . . . 146
7.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
7.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
7.3 Publications Resulting From The Thesis . . . . . . . . . . . . . . . . 150
REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
VITA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
ix
LIST OF TABLES
Table Page
3.1 Mean absolute errors for miss rate prediction . . . . . . . . . . . . . . . . . 39
4.1 Spatial resolutions and frame rates used for camera locations in GCRTA . 54
4.2 Summary of the videos used for ground truthing effort - Triskett Bridge . . 57
4.3 Summary of the videos used for ground truthing effort - West177 Tunnel . 57
4.4 Summary of testing data - Triskett Bridge . . . . . . . . . . . . . . . . . . 59
4.5 Summary of testing data - West177 Tunnel . . . . . . . . . . . . . . . . . . 59
4.6 Matching Accuracy - Triskett Bridge . . . . . . . . . . . . . . . . . . . . . 79
4.7 Matching Accuracy - West117 Tunnel . . . . . . . . . . . . . . . . . . . . . 79
5.1 Overall performance - crowdsourincg single video surveillance tasks . . . 111
5.2 Training performance - trained vs. untrained crowds . . . . . . . . . . . 112
5.3 Tasks summary - trained vs. untrained crowds . . . . . . . . . . . . . . . 113
5.4 Results - trained vs. untrained crowds . . . . . . . . . . . . . . . . . . . 113
5.5 Crowd flow estimation error rate with no crowdsourcing under video qual-
ity degradation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
5.6 Crowd flow estimation error rates using the second type of uncertainty
characterization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
5.7 Two approaches to aggregate crowdsourcing output and its impact on
crowd flow estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
5.8 Crowdsourcing for forensic analysis of surveillance video - investigation
details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
5.9 Crowdsourcing for forensic analysis of surveillance video - human detection
results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
5.10 Crowdsourcing for forensic analysis of surveillance video - human re-identification
- automatic method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
5.11 Crowdsourcing for forensic analysis of surveillance video - human re-identification
- using crowd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
x
LIST OF FIGURES
Figure Page
1.1 CPD district 7 borders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Police laptop connected to LTE using a USB dongle . . . . . . . . . . . . . 4
1.3 Locations used for subjective measurements within CPD district 7 . . . . . 5
1.4 Clarity, fluidity and video data rate along the cell edges - RTVI . . . . . . 6
1.5 Subject intrinsic variations and person re-identification challenges . . . . . 7
1.6 Sample sequences from two different cameras . . . . . . . . . . . . . . . . . 8
2.1 Miss rate versus FPPI for the RPN pedestrian detector . . . . . . . . . . . 22
2.2 Video compression and RPN performance . . . . . . . . . . . . . . . . . . 23
2.3 RPN detectors trained using QP {25,35,47,55} . . . . . . . . . . . . . . . . 24
2.4 RPN detectors trained using QP {20,35,47} . . . . . . . . . . . . . . . . . 25
2.5 Proposed Two-Stage Neural Network Block Diagram . . . . . . . . . . . . 26
2.6 Class assignments for training the Inception-v3 quality estimation network 27
2.7 Classification error rate as a function of the temporal window size k . . . . 28
2.8 Bank of RPN detectors performance . . . . . . . . . . . . . . . . . . . . . 30
3.1 ACF and LDCF miss rate versus QP . . . . . . . . . . . . . . . . . . . . . 35
3.2 Block diagram of accuracy prediction for pedestrian detection . . . . . . . 36
3.3 Prediction results for ACF and LDCF pedestrian detectors . . . . . . . . . 40
3.4 Prediction results for HOG and DPM pedestrian detectors . . . . . . . . . 41
3.5 Mean absolute error as a function of k . . . . . . . . . . . . . . . . . . . . 43
3.6 Mean absolute error for various number of responses . . . . . . . . . . . . 43
4.1 Subject intrinsic variations and person re-identification challenges . . . . . 45
4.2 Filters used to model texture . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.3 GCRTA system map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.4 Camera views at Brookpark, Puritas and Triskett sites . . . . . . . . . . . 55