Persistent Aerial Tracking Thesis by Matthias Mueller In Partial Fulfillment of the Requirements For

86 Pages·2016·21.89 MB·English

Persistent Aerial Tracking
Thesis by
Matthias Mueller
In Partial Fulfillment of the Requirements
For the Degree of
Masters of Science
in Electrical Engineering
King Abdullah University of Science and Technology, Thuwal,
Kingdom of Saudi Arabia
'April 13th, 2016
Matthias Mueller
All Rights Reserved
2
The thesis of Matthias Mueller is approved by the examination committee
Committee Chairperson: Bernard Ghanem
Committee Member: Jeff Shamma
Committee Member: Peter Wonka
3
ABSTRACT
Peristent Aerial Tracking
Matthias Mueller
In this thesis, we propose a new aerial video dataset and benchmark for low altitude
UAV target tracking, as well as, a photo-realistic UAV simulator that can be coupled
with tracking methods. Our benchmark provides the first evaluation of many state-
of-the-art and popular trackers on 123 new and fully annotated HD video sequences
captured from a low-altitude aerial perspective. Among the compared trackers, we
determine which ones are the most suitable for UAV tracking both in terms of track-
ing accuracy and run-time. We also present a simulator that can be used to evaluate
tracking algorithms in real-time scenarios before they are deployed on a UAV ”in
the field”, as well as, generate synthetic but photo-realistic tracking datasets with
free ground truth annotations to easily extend existing real-world datasets. Both the
benchmark and simulator will be made publicly available to the vision community to
further research in the area of object tracking from UAVs. Additionally, we propose a
persistent, robust and autonomous object tracking system for unmanned aerial vehi-
cles (UAVs) called Persistent Aerial Tracking (PAT). A computer vision and control
strategy is applied to a diverse set of moving objects (e.g. humans, animals, cars,
boats, etc.) integrating multiple UAVs with a stabilized RGB camera. A novel strat-
egy is employed to successfully track objects over a long period, by ’handing over the
camera’ from one UAV to another. We integrate the complete system into an off-
4
the-shelf UAV, and obtain promising results showing the robustness of our solution
in real-world aerial scenarios.
Keywords: Aerial Surveillance, Object Tracking, Autonomous UAV Control,
Dataset, Benchmark, Simulator, Unreal Engine 4
5
ACKNOWLEDGEMENTS
First and foremost, I would like to express my gratitude to my Lord and Saviour
Jesus Christ for giving my life a purpose and blessing me with a great family, friends
and talents enabling me to do this work.
I would also like to express my sincere appreciation and gratitude to KAUST for
the financial support, and to my supervisor Prof. Bernard Ghanem for his academic
support and guidance. He has been a great source of knowledge and has inspired me
with valuable ideas.
Furthermore, I extend my deepest appreciation to Neil Smith for his patience, encour-
agement, continued support and outstanding work on the simulator. He was always
ready to help with technical details and willing to stay up all night with me during
crunch times. In addition, I would like to thank Luca Passone, Ryan Boekeloo and
Mohammed Karkadan from FalconViz for risking their lives by helping me with the
real-world tests of the tracking system. I would also like to acknowledge the work of
Gopal Sharma on the reinitialization module.
Moreover, I would like to thank Adel Bibi and the rest of the IVUL group as well as
my other colleagues and friends at King Abdullah University of Science and Technol-
ogy, for their encouragement, support and all the fun times.
I am also very grateful to Prof. Jeff Shamma and Prof. Peter Wonka for being part
of the Master Thesis committee.
Finally, I would also like to thank my wife for her continued patience, support and
encouragement. I want to extend my deepest love and gratitude to her, my two
beautiful daughters and my parents by dedicating this work to them.
6
TABLE OF CONTENTS
Examination Committee Approval 2
Abstract 3
Acknowledgements 5
List of Abbreviations 9
List of Figures 10
List of Tables 12
1 Introduction 13
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.2.1 UAV Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.2.2 UAV Simulator . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.2.3 Generic Object Tracking . . . . . . . . . . . . . . . . . . . . . 18
1.2.4 UAV Tailored Tracking . . . . . . . . . . . . . . . . . . . . . . 19
1.3 Outline of the thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2 Benchmark - Offline Evaluation 22
2.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.1.1 Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.1.2 Acquisition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.1.3 Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.1.4 Long-term tracking . . . . . . . . . . . . . . . . . . . . . . . . 26
2.2 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.3 Evaluation Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.4 Experiments - Benchmark Evaluation . . . . . . . . . . . . . . . . . . 30
2.4.1 Overall Performance . . . . . . . . . . . . . . . . . . . . . . . 30
7
2.4.2 Per-Attribute Performance . . . . . . . . . . . . . . . . . . . . 31
2.4.3 Per-Video Performance . . . . . . . . . . . . . . . . . . . . . . 35
2.4.4 Sensitivity to Frame Rate . . . . . . . . . . . . . . . . . . . . 36
2.4.5 Long-Term Tracking . . . . . . . . . . . . . . . . . . . . . . . 37
2.4.6 Spatial Robustness . . . . . . . . . . . . . . . . . . . . . . . . 38
2.4.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3 Simulator - Online Evaluation 40
3.1 Simulator Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.1.1 UAV Physics Simulation and Control . . . . . . . . . . . . . . 41
3.1.2 Frame Capture, Object Segmentation, Flight Logging . . . . . 42
3.1.3 MATLAB/C++ Integration . . . . . . . . . . . . . . . . . . . 42
3.1.4 Visual Servoing . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.2 Evaluation Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.2.1 Qualitative Tracker Performance Visualization . . . . . . . . . 44
3.3 Experiments - Simulator Evaluation . . . . . . . . . . . . . . . . . . . 45
3.3.1 Sensitivity to Frame Rate . . . . . . . . . . . . . . . . . . . . 47
3.3.2 Sensitivity to UAV Response . . . . . . . . . . . . . . . . . . . 47
3.3.3 Long-Term Tracking . . . . . . . . . . . . . . . . . . . . . . . 48
3.3.4 Per-Attribute Performance . . . . . . . . . . . . . . . . . . . . 48
3.3.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4 Integration - Aerial Tracking System 51
4.1 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.2 Onboard Visual Tracking System . . . . . . . . . . . . . . . . . . . . 53
4.3 Tracking Control System . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.4 Handover System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.5 Tracker Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.5.1 Evaluation Setup . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.5.2 Evaluation Results . . . . . . . . . . . . . . . . . . . . . . . . 60
4.5.3 Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.6 Target Initialization Module . . . . . . . . . . . . . . . . . . . . . . . 65
4.7 Bounding Box Evaluation and Re-initialization Module . . . . . . . . 67
4.8 Experiments - System Evaluation . . . . . . . . . . . . . . . . . . . . 70
4.8.1 Regular Motion . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.8.2 Rapid/Erratic Motion . . . . . . . . . . . . . . . . . . . . . . 71
4.8.3 Robustness to Occlusion . . . . . . . . . . . . . . . . . . . . . 71
8
4.8.4 Camera Handover . . . . . . . . . . . . . . . . . . . . . . . . . 72
4.8.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
5 Conclusion 73
5.1 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
References 76
Appendix 86
9
LIST OF ABBREVIATIONS
FCS Flight Control System
GCS Ground Control Station
OPE One Pass Evaluation
OTB Online Tracking Benchmark
PAT Persistent Aerial Tracking
SRE Spatial Robustness Evaluation
TRE Temporal Robustness Evaluation
UAV Unmanned Aerial Vehicle
UE4 Unreal Engine 4
10
LIST OF FIGURES
2.1 Histogram of aspect ratio, bounding box size and sequence length
across three datasets: OTB100, TC128, and UAV123 (ours). . . . . . 24
2.2 First frame of selected sequences from UAV123 dataset. . . . . . . . . 25
2.3 Attribute distribution across the UAV123 dataset and a comparison of
key attributes with OTB100. . . . . . . . . . . . . . . . . . . . . . . . 27
2.4 Dataset Difficulty Comparison. . . . . . . . . . . . . . . . . . . . . . 28
2.5 Precision and success plots for OPE on UAV123. . . . . . . . . . . . . 31
2.6 Precision and success plots for OPE on OTB100. . . . . . . . . . . . 31
2.7 Precision and success plots for OPE on TC128. . . . . . . . . . . . . 31
2.8 OPE precision plots by attribute. . . . . . . . . . . . . . . . . . . . . 32
2.9 OPE success plots by attribute. . . . . . . . . . . . . . . . . . . . . . 33
2.10 OPE Precision (top) and Success (bottom) per video for UAV123. . . 35
2.11 OPE Precision (top) and Success (bottom) per video for OTB100. . . 36
2.12 Precision and success plots for OPE on UAV123@10fps. . . . . . . . . 37
2.13 Precision and success plots for OPE on UAV20L. . . . . . . . . . . . 38
2.14 Precision and success plots for SRE on UAV123. . . . . . . . . . . . . 38
3.1 Third person view of simulator environment. . . . . . . . . . . . . . . 41
3.2 Synthetic dataset generation and online tracker evaluation using the
proposed simulator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.3 Third person view of object being tracked by multiple UAVs corre-
sponding to different trackers. . . . . . . . . . . . . . . . . . . . . . . 45
3.4 Trajectory of tracker controlled UAV over the period of the simulation
andmultipletrackersboundingboxeslayeredoverthetrackedsynthetic
frame. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.1 The two classes of unmanned aerial vehicles integrated with the on-
board vision based flight control system. . . . . . . . . . . . . . . . . 52
4.2 Hardware and software of FCS and GCS. . . . . . . . . . . . . . . . . 53
4.3 Flowchart of the onboard visual tracking system. . . . . . . . . . . . 55