Computer Vision and Imaging in Intelligent Transportation Systems

411 Pages·2017·9.517 MB·English

by  Robert P. Loce, Raja Bala, Mohan Trivedi (eds.)

Computer Vision and Imaging in Intelligent
Transportation Systems
Computer Vision and Imaging in Intelligent
Transportation Systems
Edited by
Robert P. Loce
Conduent Labs, Webster, NY, USA
Raja Bala
Samsung Research America, Richardson, TX, USA
Mohan Trivedi
University of California, San Diego, CA, USA
This edition first published 2017
© 2017 John Wiley & Sons Ltd
Registered Offices
John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, USA
John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, United Kingdom
For details of our global editorial offices, for customer services and for information about how to apply for permission to
reuse the copyright material in this book please see our website at www.wiley.com.
The right of the author to be identified as the author of this work has been asserted in accordance with the Copyright,
Designs and Patents Act 1988.
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form
or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by the UK Copyright,
Designs and Patents Act 1988, without the prior permission of the publisher.
Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in
electronic books.
Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and
product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective
owners. The publisher is not associated with any product or vendor mentioned in this book.
Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this
book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this
book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. It is sold on the
understanding that the publisher is not engaged in rendering professional services and neither the publisher nor the author
shall be liable for damages arising herefrom. If professional advice or other expert assistance is required, the services of a
competent professional should be sought
Library of Congress Cataloging‐in‐Publication data applied for
ISBN: 9781118971604
A catalogue record for this book is available from the British Library.
Cover design: Wiley
Cover image: ©Anouar Akrouh / Eyeem / Gettyimages
Setin 10/12pt Warnock by SPi Global, Pondicherry, India
10 9 8 7 6 5 4 3 2 1
v
Contents
List of Contributors xiii
Preface xvii
Acknowledgments xxi
About the Companion Website xxiii
1 Introduction 1
Raja Bala and Robert P. Loce
1.1 Law Enforcement and Security 1
1.2 Efficiency 4
1.3 Driver Safety and Comfort 5
1.4 A Computer Vision Framework for Transportation Applications 7
1.4.1 Image and Video Capture 8
1.4.2 Data Preprocessing 8
1.4.3 Feature Extraction 9
1.4.4 Inference Engine 10
1.4.5 Data Presentation and Feedback 11
References 12
Part I Imaging from the Roadway Infrastructure 15
2 Automated License Plate Recognition 17
Aaron Burry and Vladimir Kozitsky
2.1 Introduction 17
2.2 Core ALPR Technologies 18
2.2.1 License Plate Localization 19
2.2.1.1 Color‐Based Methods 20
2.2.1.2 Edge‐Based Methods 20
2.2.1.3 Machine Learning–Based Approaches 23
2.2.2 Character Segmentation 24
2.2.2.1 Preprocessing for Rotation, Crop, and Shear 25
2.2.2.2 Character‐Level Segmentation 28
2.2.3 Character Recognition 28
2.2.3.1 Character Harvesting and Sorting 30
vi Contents
2.2.3.2 Data Augmentation 31
2.2.3.3 Feature Extraction 32
2.2.3.4 Classifiers and Training 34
2.2.3.5 Classifier Evaluation 37
2.2.4 State Identification 38
References 42
3 Vehicle Classification 47
Shashank Deshpande, Wiktor Muron and Yang Cai
3.1 Introduction 47
3.2 Overview of the Algorithms 48
3.3 Existing AVC Methods 48
3.4 LiDAR Imaging‐Based 49
3.4.1 LiDAR Sensors 49
3.4.2 Fusion of LiDAR and Vision Sensors 50
3.5 Thermal Imaging‐Based 53
3.5.1 Thermal Signatures 53
3.5.2 Intensity Shape‐Based 56
3.6 Shape‐ and Profile‐Based 58
3.6.1 Silhouette Measurements 60
3.6.2 Edge‐Based Classification 65
3.6.3 Histogram of Oriented Gradients 67
3.6.4 Haar Features 68
3.6.5 Principal Component Analysis 69
3.7 Intrinsic Proportion Model 72
3.8 3D Model‐Based Classification 74
3.9 SIFT‐Based Classification 74
3.10 Summary 75
References 75
4 Detection of Passenger Compartment Violations 81
Orhan Bulan, Beilei Xu, Robert P. Loce and Peter Paul
4.1 Introduction 81
4.2 Sensing within the Passenger Compartment 82
4.2.1 Seat Belt Usage Detection 82
4.2.2 Cell Phone Usage Detection 83
4.2.3 Occupancy Detection 83
4.3 Roadside Imaging 84
4.3.1 Image Acquisition Setup 84
4.3.2 Image Classification Methods 85
4.3.2.1 Windshield and Side Window Detection from HOV/HOT Images 86
4.3.2.2 Image Classification for Violation Detection 90
4.3.3 Detection‐Based Methods 94
4.3.3.1 Multiband Approaches for Occupancy Detection 94
4.3.3.2 Single Band Approaches 95
References 96
Contents vii
5 Detection of Moving Violations 101
Wencheng Wu, Orhan Bulan, Edgar A. Bernal and Robert P. Loce
5.1 Introduction 101
5.2 Detection of Speed Violations 101
5.2.1 Speed Estimation from Monocular Cameras 102
5.2.2 Speed Estimation from Stereo Cameras 108
5.2.2.1 Depth Estimation in Binocular Camera Systems 109
5.2.2.2 Vehicle Detection from Sequences of Depth Maps 110
5.2.2.3 Vehicle Tracking from Sequences of Depth Maps 113
5.2.2.4 Speed Estimation from Tracking Data 114
5.2.3 Discussion 115
5.3 Stop Violations 115
5.3.1 Red Light Cameras 115
5.3.1.1 RLCs, Evidentiary Systems 116
5.3.1.2 RLC, Computer Vision Systems 118
5.3.2 Stop Sign Enforcement Systems 123
5.4 Other Violations 125
5.4.1 Wrong‐Way Driver Detection 125
5.4.2 Crossing Solid Lines 126
References 126
6 Traffic Flow Analysis 131
Rodrigo Fernandez, Muhammad Haroon Yousaf, Timothy J. Ellis, Zezhi Chen
and Sergio A. Velastin
6.1 What is Traffic Flow Analysis? 131
6.1.1 Traffic Conflicts and Traffic Analysis 131
6.1.2 Time Observation 132
6.1.3 Space Observation 133
6.1.4 The Fundamental Equation 133
6.1.5 The Fundamental Diagram 133
6.1.6 Measuring Traffic Variables 134
6.1.7 Road Counts 135
6.1.8 Junction Counts 135
6.1.9 Passenger Counts 136
6.1.10 Pedestrian Counts 136
6.1.11 Speed Measurement 136
6.2 The Use of Video Analysis in Intelligent Transportation Systems 137
6.2.1 Introduction 137
6.2.2 General Framework for Traffic Flow Analysis 137
6.2.2.1 Foreground Estimation/Segmentation 139
6.2.2.2 Segmentation 140
6.2.2.3 Shadow Removal 140
6.2.2.4 Morphological Operations 141
6.2.2.5 Approaches Based on Object Recognition 141
6.2.2.6 Interest‐Point Feature Descriptors 141
6.2.2.7 Appearance Shape–Based Descriptors 142
viii Contents
6.2.2.8 Classification 142
6.2.2.9 Analysis 143
6.2.3 Application Domains 143
6.3 Measuring Traffic Flow from Roadside CCTV Video 144
6.3.1 Video Analysis Framework 144
6.3.2 Vehicle Detection 146
6.3.3 Background Model 146
6.3.4 Counting Vehicles 149
6.3.5 Tracking 150
6.3.6 Camera Calibration 150
6.3.7 Feature Extraction and Vehicle Classification 152
6.3.8 Lane Detection 153
6.3.9 Results 155
6.4 Some Challenges 156
References 159
7 Intersection Monitoring Using Computer Vision Techniques for Capacity,
Delay, and Safety Analysis 163
Brendan Tran Morris and Mohammad Shokrolah Shirazi
7.1 Vision‐Based Intersection Analysis: Capacity, Delay, and Safety 163
7.1.1 Intersection Monitoring 163
7.1.2 Computer Vision Application 164
7.2 System Overview 165
7.2.1 Tracking Road Users 166
7.2.2 Camera Calibration 169
7.3 Count Analysis 171
7.3.1 Vehicular Counts 171
7.3.2 Nonvehicular Counts 173
7.4 Queue Length Estimation 173
7.4.1 Detection‐Based Methods 174
7.4.2 Tracking‐Based Methods 175
7.5 Safety Analysis 177
7.5.1 Behaviors 178
7.5.1.1 Turning Prediction 179
7.5.1.2 Abnormality Detection 179
7.5.1.3 Pedestrian Crossing Violation 179
7.5.1.4 Pedestrian Crossing Speed 181
7.5.1.5 Pedestrian Waiting Time 182
7.5.2 Accidents 182
7.5.3 Conflicts 185
7.6 Challenging Problems and Perspectives 187
7.6.1 Robust Detection and Tracking 187
7.6.2 Validity of Prediction Models for Conflict and Collisions 188
7.6.3 Cooperating Sensing Modalities 189
7.6.4 Networked Traffic Monitoring Systems 189
7.7 Conclusion 189
References 190
Contents ix
8 Video‐Based Parking Management 195
Oliver Sidla and Yuriy Lipetski
8.1 Introduction 195
8.2 Overview of Parking Sensors 197
8.3 Introduction to Vehicle Occupancy Detection Methods 200
8.4 Monocular Vehicle Detection 200
8.4.1 Advantages of Simple 2D Vehicle Detection 200
8.4.2 Background Model–Based Approaches 200
8.4.3 Vehicle Detection Using Local Feature Descriptors 202
8.4.4 Appearance‐Based Vehicle Detection 203
8.4.5 Histograms of Oriented Gradients 204
8.4.6 LBP Features and LBP Histograms 207
8.4.7 Combining Detectors into Cascades and Complex Descriptors 208
8.4.8 Case Study: Parking Space Monitoring Using a Combined
Feature Detector 208
8.4.9 Detection Using Artificial Neural Networks 211
8.5 Introduction to Vehicle Detection with 3D Methods 213
8.6 Stereo Vision Methods 215
8.6.1 Introduction to Stereo Methods 215
8.6.2 Limits on the Accuracy of Stereo Reconstruction 216
8.6.3 Computing the Stereo Correspondence 217
8.6.4 Simple Stereo for Volume Occupation Measurement 218
8.6.5 A Practical System for Parking Space Monitoring Using a Stereo System 218
8.6.6 Detection Methods Using Sparse 3D Reconstruction 220
Acknowledgment 223
References 223
9 Video Anomaly Detection 227
Raja Bala and Vishal Monga
9.1 Introduction 227
9.2 Event Encoding 228
9.2.1 Trajectory Descriptors 229
9.2.2 Spatiotemporal Descriptors 231
9.3 Anomaly Detection Models 233
9.3.1 Classification Methods 233
9.3.2 Hidden Markov Models 234
9.3.3 Contextual Methods 234
9.4 Sparse Representation Methods for Robust Video Anomaly 
Detection 236
9.4.1 Structured Anomaly Detection 237
9.4.1.1 A Joint Sparsity Model for Anomaly Detection 238
9.4.1.2 Supervised Anomaly Detection as Event Classification 242
9.4.1.3 Unsupervised Anomaly Detection via Outlier Rejection 242
9.4.2 Unstructured Video Anomaly Detection 243
9.4.3 Experimental Setup and Results 245
9.4.3.1 Anomaly Detection in Structured Scenarios 246
9.4.3.2 Detection Rates for Single‐Object Anomaly Detection 246
x Contents
9.4.3.3 Detection Rates for Multiple‐Object Anomaly Detection 246
9.4.3.4 Anomaly Detection in Unstructured Scenarios 250
9.5 Conclusion and Future Research 253
References 254
Part II Imaging from and within the Vehicle 257
10 Pedestrian Detection 259
Shashank Deshpande and Yang Cai
10.1 Introduction 259
10.2 Overview of the Algorithms 259
10.3 Thermal Imaging 260
10.4 Background Subtraction Methods 261
10.4.1 Frame Subtraction 261
10.4.2 Approximate Median 262
10.4.3 Gaussian Mixture Model 263
10.5 Polar Coordinate Profile 263
10.6 Image‐Based Features 265
10.6.1 Histogram of Oriented Gradients 265
10.6.2 Deformable Parts Model 266
10.6.3 LiDAR and Camera Fusion–Based Detection 266
10.7 LiDAR Features 268
10.7.1 Preprocessing Module 268
10.7.2 Feature Extraction Module 268
10.7.3 Fusion Module 268
10.7.4 LIPD Dataset 270
10.7.5 Overview of the Algorithm 270
10.7.6 LiDAR Module 272
10.7.7 Vision Module 275
10.7.8 Results and Discussion 276
10.7.8.1 LiDAR Module 276
10.7.8.2 Vision Module 276
10.8 Summary 280
References 280
11 Lane Detection and Tracking Problems in Lane Departure Warning Systems 283
Gianni Cario, Alessandro Casavola and Marco Lupia
11.1 Introduction 283
11.1.1 Basic LDWS Algorithm Structure 284
11.2 LD: Algorithms for a Single Frame 285
11.2.1 Image Preprocessing 285
11.2.1.1 Gray‐Level Optimization 286
11.2.1.2 Image Smoothing 286
11.2.2 Edge Extraction 287
11.2.2.1 Second‐Order Derivative Operators 288
11.2.2.2 Canny’s Algorithm 290
Contents xi
11.2.2.3 Comparison of Edge‐Detection Algorithms 291
11.2.3 Stripe Identification 291
11.2.3.1 Edge Distribution Function 292
11.2.3.2 Hough Transform 292
11.2.4 Line Fitting 294
11.2.4.1 Linear Fitting 295
11.2.4.2 LP Fitting 295
11.3 LT Algorithms 297
11.3.1 Recursive Filters on Subsequent N frames 298
11.3.2 Kalman Filter 298
11.4 Implementation of an LD and LT Algorithm 299
11.4.1 Simulations 300
11.4.2 Test Driving Scenario 300
11.4.3 Driving Scenario: Lane Departures at Increasing Longitudinal Speed 300
11.4.4 The Proposed Algorithm 302
11.4.5 Conclusions 303
References 303
12 Vision‐Based Integrated Techniques for Collision Avoidance Systems 305
Ravi Satzoda and Mohan Trivedi
12.1 Introduction 305
12.2 Related Work 307
12.3 Context Definition for Integrated Approach 307
12.4 ELVIS: Proposed Integrated Approach 308
12.4.1 Vehicle Detection Using Lane Information 309
12.4.2 Improving Lane Detection using On‐Road Vehicle Information 312
12.5 Performance Evaluation 313
12.5.1 Vehicle Detection in ELVIS 313
12.5.1.1 Accuracy Analysis 313
12.5.1.2 Computational Efficiency 314
12.5.2 Lane Detection in ELVIS 316
12.6 Concluding Remarks 319
References 319
13 Driver Monitoring 321
Raja Bala and Edgar A. Bernal
13.1 Introduction 321
13.2 Video Acquisition 322
13.3 Face Detection and Alignment 323
13.4 Eye Detection and Analysis 325
13.5 Head Pose and Gaze Estimation 326
13.5.1 Head Pose Estimation 326
13.5.2 Gaze Estimation 328
13.6 Facial Expression Analysis 332
13.7 Multimodal Sensing and Fusion 334
13.8 Conclusions and Future Directions 336
References 337