Advanced Driver Intention Inference: Theory and Design PDF
Preview Advanced Driver Intention Inference: Theory and Design
Advanced Driver Intention Inference Theory and Design YANG XING, PHD Research Fellow School of Mechanical and Aerospace Engineering Nanyang Technological University Singapore CHEN LV Assistant Professor School of Mechanical and Aerospace Engineering Nanyang Technological University Singapore DONGPU CAO Canada Research Chair in Driver Cognition and Automated Driving Department of Mechanical and Mechatronics Engineering University of Waterloo Canada ] Elsevier Radarweg29,POBox211,1000AEAmsterdam,Netherlands TheBoulevard,LangfordLane,Kidlington,OxfordOX51GB,UnitedKingdom 50HampshireStreet,5thFloor,Cambridge,MA02139,UnitedStates Copyright(cid:1)2020ElsevierInc.Allrightsreserved. Nopartofthispublicationmaybereproducedortransmittedinanyformorbyanymeans,electronicor mechanical,includingphotocopying,recording,oranyinformationstorageandretrievalsystem,without permissioninwritingfromthepublisher.Detailsonhowtoseekpermission,furtherinformationaboutthe Publisher’spermissionspoliciesandourarrangementswithorganizationssuchastheCopyrightClearance CenterandtheCopyrightLicensingAgency,canbefoundatourwebsite:www.elsevier.com/permissions. ThisbookandtheindividualcontributionscontainedinitareprotectedundercopyrightbythePublisher(other thanasmaybenotedherein). Notices Knowledgeandbestpracticeinthisfieldareconstantlychanging.Asnewresearchandexperiencebroadenour understanding,changesinresearchmethods,professionalpractices,ormedicaltreatmentmaybecome necessary. Practitionersandresearchersmustalwaysrelyontheirownexperienceandknowledgeinevaluatingandusing anyinformation,methods,compounds,orexperimentsdescribedherein.Inusingsuchinformationor methodstheyshouldbemindfuloftheirownsafetyandthesafetyofothers,includingpartiesforwhomthey haveaprofessionalresponsibility. Tothefullestextentofthelaw,neitherthePublishernortheauthors,contributors,oreditors,assumeany liabilityforanyinjuryand/ordamagetopersonsorpropertyasamatterofproductsliability,negligence orotherwise,orfromanyuseoroperationofanymethods,products,instructions,orideascontainedinthe materialherein. LibraryofCongressCataloging-in-PublicationData AcatalogrecordforthisbookisavailablefromtheLibraryofCongress BritishLibraryCataloguing-in-PublicationData AcataloguerecordforthisbookisavailablefromtheBritishLibrary ISBN:978-0-12-819113-2 ForinformationonallElsevierpublicationsvisitourwebsiteat https://www.elsevier.com/books-and-journals Publisher:MatthewDeans AcquisitionsEditor:CarrieBolger EditorialProjectManager:GabrielaD.Capille ProductionProjectManager:SreejithViswanathan Coverdesigner:AlanStudholme TypesetbyTNQTechnologies List of Abbreviations GP Gaussian Process ABS AntilockBrakingSystem HMM Hidden MarkovModel ACC AdaptiveCruiseControl HHMM Hierarchical HiddenMarkovModel ACP ArtificialSociety,Computational HOG Histogramof OrientedGradients Experiments,andParallelExecution HRI Human-RobotInteraction ACT-R AdaptiveControlofThought-Rational HT Hough Transform ADAS AdvancedDriverAssistanceSystem IDDM Intention-driven DynamicModel A/D Analog/Digital IMM InteractiveMultiple Model AIOHMM AutoregressiveInput-Output HMM IMU Inertial MeasurementUnit ANN ArtificialNeural Network IOHMM Input-Output HiddenMarkovModel AVM AroundViewMonitoring IPM Inverse PerspectiveMapping BCI Brain-ComputerInterface JTSM JointTime-Series Modelling BF BayesianFilter KRR KernelRidgeRegression BN BayesianNetwork LADAR Laser DetectionandRanging CAN ControllerAreaNetwork LANA Lane FindinginAnother Domain CBN CausalBayesian Network LCII Lane ChangeIntentionInference CDI ComprehensiveDecision Index LDA Lane DepartureAvoidance CHMM ContinuousHidden MarkovModel LDW Lane DepartureWarning CLNF Conditional LocalNeuralFields LIDAR Light DetectionandRanging CLR ConstantLearningRate LKA Lane KeepingAssistance CPS Cyber-PhysicalSpace LOO Leave-One-Out CPSS Cyber-Physical-Social Space LSV Lane SamplingandVoting CNN ConvolutionalNeural Network LSTM LongShort-TermMemory CSI ChannelStateInformation MIC Maximal Information Coefficient DBSCAN Density-based Spatial Clustering of MHMM ModifiedHidden MarkovModel ApplicationwithNoise MLR Multisteps LearningRate DBN DynamicBayesianNetwork NGSIM Next-Generation Simulation DII DriverIntentionInference NFIR NonlinearFiniteImpulseResponse DWT Discrete WaveletTransform NN Neural Networks ED EdgeDistribution OLS Orthogonal LeastSquares EEG Electroencephalograph OOB Out-of-Bag ECG Electrocardiogram PCA Principal ComponentAnalysis EMG Electromyography PRM Revolutions PerMinute EOG Electrooculography RANSAC RandomSampleConsensus ERRC ErrorReduction Ratio-Causality RF RandomForest EV Electric Vehicle RGB Red-Green-Blue FFNN Feedforward NeuralNetwork RGB-D Red-Green-Blue Depth FIR FieldImpedance Equalizer RMSE RootMeanSquareError FPR False-Positive Rate RNN Recurrent NeuralNetwork GA GeneticAlgorithm ROC ReceiverOperating Characteristic GAN GenerativeAdversarialNetworks ROI RegionofInterest GMM GaussianMixture Model RVM RelevanceVectorMachine GNSS GlobalNavigation SatelliteSystem SA Situation Awareness GOLD GenericObstacleandLaneDetection SAE Society ofAutomobileEngineers GPS GlobalPositioning System SBL Sparse BayesianLearning v vi LISTOFABBREVIATIONS SF SteerableFilter TS TimeSliced SCR SkinConductance Response TPR True-Positive Rate SWA SideWarning Assistance TTI TimetoIntersection SVM SupportVectorMachine TTC TimetoCollision SVR SupportVectorRegression TTCCP TimetoCriticalProbability TDV Traffic-Driver-Vehicle V2V Vehicle-to-Vehicle THW TimeHeadway WHO World HealthOrganization Abstract Longitudinal and lateral control of the vehicle on the inference.Chapters3and4containthetechniquesfor highwayarehighlyinteractivetasksforhumandrivers. traffic context perception that focus on sensor integra- The intelligent vehicles and the advanced driver- tion, sensor fusion, and road perception. A review of assistancesystems(ADAS)needtohaveproperaware- lanedetectiontechniquesanditsintegrationwithapar- nessofthetrafficcontextaswellasthedrivertomake alleldrivingframeworkisproposed.Next,anovelinte- an appropriate assistant to the driver. The ADAS also grated lane detection system is designed. Chapters 5 needtounderstandthepotentialdriverintentcorrectly and6discussthedriverbehaviorissues,whichprovide since it shares the control authority with the human the driver behavior monitoring system for normal driver.Thisbookprovidesresearchonthedriverinten- driving and secondary tasks detection. The first part is tion inference, particular focus on the two typical based on the conventional feature selection method, vehicle control maneuvers, namely, lane change ma- while the second part introduces an end-to-end deep neuver and braking maneuver on highway scenarios. learning framework. Understanding the driver status A primarymotivation of thisbook is topropose algo- and behaviors is the preliminary task for driver inten- rithms that can accurately model the driver intention tioninference.Thedesignandanalysisofdriverbraking inferenceprocess.Driver’sintentionwillberecognized and lane change intention inference systems are pro- based on the machine learning methods due to its posed in Chapters 7 and 8. Machine-learning models goodreasoningandgeneralizingcharacteristics.Sensors and time-series deep-learning models are used to esti- inthevehicleareusedtocapturecontexttrafficinforma- mate the longitudinal and lateral driver intention. tion, vehicle dynamics, and driver behavior Finally, discussions and conclusions are made in information. Chapter9. This book is organized in sections and chapters, whereeachchapteriscorrespondingtoaspecifictopic. KEYWORDS Chapter1introducesthemotivation,humanintention background, and general methodological framework ADAS,Computervision,Driverbehaviors,Driverinten- usedinthisbook.Chapter2includestheliteraturesur- tion inference, Intelligent vehicles,Automated driving, veyandthestate-of-the-artanalysisofdriverintention Machinelearning. vii CHAPTER 1 Introduction Worldwidetrafficdepartmentshavereportedthatmore ADAS techniques such as lane departure avoidance, than 1.2 million traffic-related injuries happen each lane keeping assistance, and side warning assistance year. Among these traffic accidents, more than 80% can help the driver to make the right decision and werecausedbyhumanerrors[1].TheWorldHealthOr- reducetheworkload. ganization (WHO) reported that traffic accidents each It is predicted that the shipment of ADAS in the year cost around V518 billion worldwide and on future has great potential and can generate a huge average, 1%e2% of the world gross domestic product amountofcommercialbenefitbasedonmanyautomo- [2,3]. In the past, in-vehicle passive safety systems tivemarketanalyzers.OneexampleisshowninFig.1.2 suchasairbagsandseatbeltshaveplayedasignificant accordingtothepredictionofGrandViewResearch,Inc. role in the protection of drivers and passengers. These ADAS products will show a significant increase in the technologies have saved millions of lives. However, next5years.ThereforetheutilizationofADASproducts they are not designed to prevent accidents from willbecomemoreaccessibletothepublic,althoughthis happening but just try to minimize the injuries after canbringaseriesofproblems.Forexample,thefinan- the accidents happen[4].Therefore recent efforts have cialcostwillincrease.Also,asmostoftheautomotive been devoted to the development of safer and intelli- companies are developing their ADAS products, safety gentsystemstowardthepreventionofaccidents.These insurance and product quality can be different from systems are known as the Advanced Driver Assistance each other. Once the drivers are getting familiar with Systems(ADAS). theseproducts,theywillheavilyrelyonthesesystems. ADAS is a series of fast-developing techniques that AveryfamousexampleistheTeslacarcrashesthatare aredesignedforimprovingdriversafetyandincreasing caused bytheirautopilotADASproducts, asshownin thedrivingexperience[5].ADASreliesonamultimodal Fig.1.3.TheautopilotproductsofTeslaareoneofthe sensor fusion technique to integrate multiple sensors mostsuccessfulcommercialdriverassistanceandsemi- suchaslightdetectionandranging(lidar),radar,cam- automated driving assistance systeminthe world. The era,andGPSintoaholisticsystem.Thesensors’work- product is set of intelligent computing, perception, ing ranges are shown in Fig. 1.1. Most of the current and control units, which can significantly increase FIG.1.1 Distribution of Advanced DriverAssistanceSystems in an advancedvehicle(deepscale.ai/adas. html).Lidar,lightdetectionandranging. AdvancedDriverIntentionInference.https://doi.org/10.1016/B978-0-12-819113-2.00001-4 Copyright©2020ElsevierInc.Allrightsreserved. 1 2 AdvancedDriverIntentionInference drivingsafetyissues.However,evensuchasmartsystem factor, the driver. Vehicles are working in a three- can be reported for car crashes worldwide. One of the dimensional environment with continuous driver- mostcommonreasonsforacrashisthedriverovertrust- vehicle-roadinteractions.Driversarethemostessential ingtheautopilotwhenthesystemisactivated,whichis partofthissystem,whocontrolthevehiclebasedonthe aprobleminthefuture. surroundingtrafficcontextperception.Thereforeallow- ThereasonswhyADAScannotbe100%trustedare ing ADAS to understand driver behaviors and follow multifold. Currently, most of the reasons are due to driver’s intention is of importance to driver safety, immaturetechniques;however,adeeperreasonisthat vehicledrivability,andtrafficefficiency. thedriverandtheautomationlackmutualunderstand- Humandriverintentioninferenceisanidealwayto ing.TheinputsofcurrentADASaremainlybasedonly allowADAStoobtaintheabilityofreasoning.Therea- onthevehicledynamicstatesandtrafficcontextinfor- sons for developing driver intention inference tech- mation. Most of the systems ignore the most critical nique are multifold: first of all, the most important FIGURE1.2 AdvancedDriverAssistanceSystemsmarketprediction(GrandViewResearch,Inc.https:// www.grandviewresearch.com).ACC,adaptivecruisecontrol;AEB,automaticemergencybraking;AFL, adaptivefrontlight;BSD,blindspotdetection;LDWS,lanedeparturewarningsystems;TPMS,tirepressure monitoringsystem. FIGURE1.3 ATeslacarhascrashedintoaparkedpolicecarinCalifornia,USA.(https://www.bbc.com/ news/technology-44300952). CHAPTER1 Introduction 3 and significant motivation is to improve driver and can act as the guidance to design an automatic vehicle safety. Accordingly, two different driving sce- decision-makingsystem. narios require inferring the driver’s intention. The first Moreover,intermsofthelevel3automatedvehicle one is to better assess the risk in the future based on (according to the SAE International standard on the the driver’s interesting region. The second one is to classification of automated vehicles), accurate driver avoidmakingdecisionsthatareoppositetothedriver’s intentionpredictionenablesasmootherandsafertran- intent. For the first case, there is evidence that a large sition between the driver and the autonomous number of accidents are caused by human error or controller[11,12].Whenthelevel3automatedvehicles misbehavior, including cognitive (47%), judgment are operating in an autonomous condition, all the (40%), and operational errors (13%) [6]. Therefore drivingmaneuversarehandledbythevehicle.However, monitoring and correcting driver intention and once the vehicle cannot deal with an emergent situa- behavior seem to be crucial in the effectiveness of a tion, it has to give the driving authority to the driver. future ADAS. Meanwhile, the increasing use of in- This process is known as disengagement [13]. In such vehicledevicesandinformationsystemstendtodistract a case, the vehicle can assess the takeover ability of thedriverfromthedrivingtask.Forthedesignoffuture thedriveraccordingtothecontinuouslydetectedinten- ADAS, it is therefore beneficial to integrate intended tion.Ifthedriverisfocusingonthedrivingtaskatthat driverbehaviorsfromtheearlydesignstages[7,8]. moment andhas an explicit intention, the vehicle can ADASusuallyautomaticallyinterveneinthevehicle warn the driver and pass the driving authority to the dynamics and share the control authority. To ensure driver.Thiswillmakesurethetransitionbetweendriver cooperation, it is crucial that ADAS is aware of driver andcontrollerisassmoothaspossible.However,ifthe intentionanddoesnotoperateagainstthedriver’swill- driver is believed to be unable to handle the driving ingness.Forexample,incomplextrafficconditionssuch task, the autonomous driving unit should help the asintersectionandroundabout,itiscrucialnottointer- driver gain situation awareness as soon as possible ruptthedrivermakingdecisions,especiallynottosus- andtakeemergencyactionimmediately. pend the driver with misleading instructions. This Anotheressentialreasontodevelopthedriverinten- makes it reasonable or even necessary for ADAS to tioninferencesystemisitwillcontributetothedevel- have the ability to accurately understand the driver’s opment of automated vehicles. As shown in Fig. 1.4, driving intention. On the other hand, intention infor- each level of understanding about the driver can be mation enables for more accurate prediction of future mapped into the corresponding intelligent level of an trajectories, which would be beneficial to risk assess- autonomousvehicle.Comprehensiveresearchoneach ment systems [9,10]. Driver intention inference will layerwillcontributetothedevelopmentoftherelative benefit the construction of the driver model, which layer in the autonomous vehicle. Driver intention FIGURE1.4 Theevolutionfromthecurrentvehicletofutureautonomousvehicle. 4 AdvancedDriverIntentionInference recognition is a relative higher-level understanding of driver behavior analysis is of importance to infer humandriversandrelatedtothedecision-makinglayer driverintention. of autonomous vehicles. Modeling driver intention 4. Driver lane change intention inference algorithms: mechanismsiscriticaltotheconstructionofautomated Based on the specific traffic context and driver be- decision-making algorithms. Human drivers are the haviors, the next task is to infer driver intention teacher of automated drivers. The automated drivers properly. The algorithms for intention inference can learn when and how to make the decision based should have the ability to capture the long-term on the driver’s intention knowledge. Once the human dependency between the temporal sequences. driverbecomesthepassengerintheautomateddriving Moreover,theintentioninferencealgorithmsshould vehicles, it will be easier for the passengers to accept predicttheintentionasearlyaspossible. that the automated driving systems is such systems The driver lane change intention platform requires that can remember the driving habit of the passenger. the integration of software and hardware systems. Therefore a good study about when and how drivers Driverintentioninferencehastotakethetrafficcontext, generate their intentions will benefit the design of the driver behaviors, or dynamic vehicle information into decision-makingmoduleforintelligentvehicles.Based consideration,whichwillfusemultimodalsignalsand on such a design method, the vehicles will be more mining the long-term dependency between different similar to human drivers, which will make it much signals based on machine learning methods. In terms easier for humans to accept these highly intelligent of the hardware system, the sensors, included in this vehicles. book, contain RGB and RGB-D cameras and vehicle As discussed earlier, teaching ADAS to understand navigation system. Besides, all the sensors are tested driver intention is essential as well as challenging to and mounted on a real vehicle in this case to collect enhance the safety of the driver-vehicle-traffic close- naturalistic data. Specifically, the traffic context such loop system. To focus more, this book will target two aslanepositionsandfrontvehiclepositionwillbepro- ofthemostpopulardrivingscenariosinbothlongitudi- cessedwithimage-processingmethods.Onewebcam- nalandlateraldirections,namely,thebrakingandlane eraismountedinsidethecabinet.Thedriverbehavior changemaneuvers.Forexample,duringastandardlane dynamics will be evaluated within a steady and dy- changemaneuver,thedriverisexpectedtoperformase- namic vehicle. The RGB-D camera (Microsoft Kinect ries of behaviors (e.g., mirror checking and turn the V2.0)willbeusedforthesteadyvehicle,whileanother steeringwheel).Thedriver’slanechangeintentioncan web camerawill beusedto recordthe driverbehavior be inferred in an early stage by recognizing driver be- during the highway driving task. These signals are haviors and traffic context situations. A driver lane recorded with one laptop for further processing and change intention system facing next-generation ADAS analyzing. The algorithms used in this project are isdevelopedinthisstudy.Basedonthis,fourmainob- mainly focused on machine learning methods, which jectivesaredetermined: include supervised learning, unsupervised learning, 1. Driver intention process analysis: To predict driver anddeeplearningmodels.Allthealgorithmsarewrit- lane change intention, it is vital to understand the teninMATLABandCþþ. human intention mechanism, such as how Thedriver’sintentioninferencetaskdescribedinthis theintentionisgeneratedandwhatisthestimuliof bookreliesonmachinelearningalgorithmstoworkin the intention. The nature behind driver intention realtime.Thereasonsforusingmachinelearningcanbe isthefirstquestionthatneedstobeanswered. multifold. First, the real-time traffic context and driver 2. Trafficcontextperception:Thedriverisinthemiddle behaviordatacanbehighdimensionalandoflargevol- ofthetraffic-driver-vehicleloop.Trafficcontextisthe ume, and very few mathematic models can deal with inputtothedriverperceptionsystem,whichmakes such data. However, machine learning algorithms are itactasthestimuliofthedriver’sintention.There- useful for high-dimensional multimodal data process- fore understanding the current traffic situation will ing.Second,theutilizationofamachinelearningalgo- benefittheintentioninferencesystem. rithm enables learning the long-term dependency 3. Understanding driver behaviors: Driver behaviors, between driver behaviors and traffic context, which such as mirror checking, are the most important significantly increases the inference accuracy for the clues before the driver makes a lane change. The lane change intention. Finally, it is hard to find the driverhastoperformaseriesofchecking actionto intention generation and inference pattern based on haveafullunderstandingofthesurroundingcontext observationandmodeling.Themachinelearningalgo- beforehe/shedecidestochangethelane.Therefore rithms provide an efficient way to learn knowledge
The list of books you might like
