Estimating the risk associated with transportation technology using multifidelity simulation

4.4 MB·

by  Erik J. Schlicht

Estimating the risk associated with transportation
technology using multifidelity simulation
ErikJ.Schlicht [email protected]
ComputationalCognitionGroup,C2-g
3432DenmarkAvenue,Suite610
St. Paul,MN55123USA
7 NicholeMorris [email protected]
1
HumanFIRSTLaboratory,UniversityofMinnesota
0
200Transportation&SafetyBuilding
2
511WashingtonAve. SE
n
UniversityofMinnesota
a
J Minneapolis,Minnesota,55455USA

1
3

] Abstract associatedwithtransportationtechnologyprior todeploy-
P ment. These simulations must include accurate quantita-
Thispaperprovidesaquantitativemethodfores-
A
tive models of all the components of the human interact-
timatingtheriskassociatedwithcandidatetrans-
. ingwiththetechnology. Forexample,inordertoevaluate
t portation technology, before it is developed and
a theriskassociatedwithcandidatecollision-avoidancetech-
t deployed. The proposed solution extends previ-
s nology in aerospace systems, high-fidelity models of the
[ ousmethodsthatrelyexclusivelyonlow-fidelity
human-in-the-loop experimental data, or high- airspaceencounters,airframedynamics,equippedsensors,
2 collision-avoidance logic and pilot decisions are required
fidelity traffic data, by adopting a multifidelity
v (Makietal.,2010;Chryssanthacopoulos&Kochenderfer,
approachthatleveragesdatafrombothlow-and
8 2011; Mueller & Kochenderfer, 2016). The latter is only
8 high-fidelity sources. The multifidelity method
affordedsincecommercialpilotsarerequiredtoadhereto
5 overcomes limitations inherent to existing ap-
collisionavoidancerecommendationsinthecaseofanear
8 proachesbyallowingamodeltobetrainedinex-
0 midaircollision. Therefore,thepilotcanbequantitatively
pensively,whilestillassuringthatitspredictions
. representedasareactiontimedistribution.
1 generalizetothereal-world. Thisallowsforcan-
0
didate technologies to be evaluated at the stage Intransportationsystemswherehumansarenotrequiredto
7
ofconception,andenablesamechanismforonly adhere to system recommendations, a quantitative model
1
: thesafestandmosteffectivetechnologytobede- of how candidate technologies impact human decision-
v
velopedandreleased. makingisnecessary. However,thehigh-fidelity(i.e.,real-
i
X world) data needed to develop these models is not avail-
r able until after the technology has already been deployed.
a 1.Introduction As a result, the ability to evaluate system safety has pri-
marily focused on post hoc analyses that exclusively rely
The integration of technology into vehicles has become
on high-fidelity data (Yang & Koutsopoulos, 1996; Aga-
exceedingly common across the transportation industry.
mennoni et al., 2012; Gindele et al., 2013; Lefevre et al.,
However,itisoftenunclearhowchangestotechnologywill
2014; Morton et al., 2016; Wheeler et al., 2016). There-
impactsubsequentsystemsafetyuntilafterthetechnology
fore, in terms of time and development costs, these data
hasbeendeveloped,deployedandwidelyadopted.
areextremelyexpensivetoobtain,andtendtoprovideonly
Thisisprimarilyduetothefactthathigh-fidelitycomputer reactionarysupporttorisksalreadyimposedonasystem.
simulationsarerequiredtoquantitativelyestimatetherisk
Anotherapproachistorelyonlow-fidelitydatathatisob-
tainedfromhuman-in-the-loopdrivingsimulation(Gruen-
ingetal.,1998;Shechtmanetal.,2009;Leeetal.,2003).
Preliminarywork. UnderreviewbytheInternationalConference
onMachineLearning(ICML). Since the driving environment and technology can be ex-
MultifidelityRiskEstimation-SubmittedtoICML2017
tweenlow-fidelityandhigh-fidelityenvironments(Schlicht
etal.,2012),providedparticipantexpertisewasheldequiv-
High Fidelity
alent.
1: Data Collection 3: Evaluate Model 4: Estimate Risk
However, the same theoretical results found that partici-
Real-World SEigxntes rOnnally )hcet | deeps(pReal-World Speed RoTeef caChla-nWnodoliordlgdayt eR isk ptrheaaenlrt-eweaxorpreledsritghisnueimficcaaannntaacddtivofefresrsree(enly.cge.is,miepnxaptchetertmapcoitdliooeltnssp)rtaaenkddeicnttihboeynstahcief-
Environment Technology tions taken by experimental participants (e.g., novice pi-
2: Train Model lots). Morespecifically,thelow-fidelitydatashouldnotbe
Simulated SEigxntes rOnnallyS+Inig-Vnes hOicnlley ygolonhceT etadidnaC)y( deepS noitidnoCBaseline Technology Sim Sypy2e1=e=fd(fx( (x)x)) uibfcasyacentedttxhtpdoaeitfprfmterseroedadnineccdltesrttheirnaoalsiK-newe-tLodarkulDdesniivnpegebrrylgfooenwrnomc-vfieaidcnbeecesleit.twiyfTedtehhanitesaraiewcstiiidslolunanessotittogapnktriehefne--
Low Fidelity dicthigh-fidelitybehavior(Schlichtetal.,2012).
This study seeks to extend these theoretical findings
Figure1.Diagramdepictingtheuseofmultifidelitymethodsfor
(Schlichtetal.,2012)byutilizingamultifidelityapproach
estimatingtheriskassociatedwithtransportationtechnology(i.e.,
toestimatetheriskassociatedwithcandidatevehicletech-
In-vehiclesign(IVS)technology)
nology. More specifically, we investigate the risk associ-
atedwithin-vehiclesignage(IVS)technologybyleverag-
ing a multifidelity predictive model in Monte-Carlo simu-
perimentallymanipulated,thesedataallowforinsightinto lation(Figure1). Thismethodallowstheriskofcandidate
how driving behavior is impacted by changes to technol- transportationtechnologytobeestimatedpriortobeingde-
ogy. Moreover, since candidate technology can be evalu- ployed, and is the first to use a multifidelity approach for
ated prior to development, it saves time and money by al- risk estimation. The next section overviews the details of
lowingforchangestotechnologytobeimplementedbefore thelow-andhigh-fidelitydatawhichwerecollectedforthis
thedevelopmentstage(Gietelinketal.,2006). effort.
A major concern with using low-fidelity simulation data
toestimatereal-worldrisk, however, isthatmodelsdevel- 2.Low-andHigh-FidelityData
oped using this data may not generalize to the real-world.
Inordertoaccuratelyestimatetheriskassociatedwithcan-
Therefore, exclusively relying on low-fidelity data to esti-
didate transportation technology, it is desirable that the
matesafetyisnotarecommendedpractice,asitmaylead
baseline condition in the low-fidelity simulation overlaps
toincorrectconclusions.
with real-world conditions (i.e., participant, environmen-
Fortunately,multifidelityapproacheshavebeendeveloped tal, and technological factors). This helps assure that the
that leverage both low- and high-fidelity data in order to model developed using low-fidelity data generalizes well
maximize the generalization of results, while minimiz- to the high-fidelity context. More specifically, it allows
ing the cost associated with estimation. Indeed, such ap- ustotrainamodelthatpredictsdrivingperformance(i.e.,
proaches have been successfully used in wing-design op- speed) under candidate technology conditions using only
timization (Robinson et al., 2006), robotic learning (Cut- low-fidelity simulation data, once the similarity between
ler et al., 2015), and have more recently been extended to performanceisquantitativelyverified.
human-in-the-loopsystems(Schlichtetal.,2012).
Once the model is trained, we can use high-fidelity data
Previous work on multifidelity models of human-in-the- as inputs to the model in order to predict real-world driv-
loop systems are theoretical and were used to predict pi- ingbehavior(i.e.,speed)undertechnologyconditionsthat
lotdecisionsduringaself-separationencounterwithanin- have not yet been deployed. Finally, the predicted speed
truderaircraft(Schlichtetal.,2012). Thesetheoreticalef- distributions can be used in a Monte-Carlo simulation to
forts found that small differences between the simulated estimatetheriskassociatedwiththenewtechnology. The
environmentandthereal-worlddonotimpactthegeneral- nextsub-sectiondetailshowthelow-fidelitydataneededto
ization of the model estimates. More specifically, differ- trainourmodelswascollected.
encesinsimulationenvironmentresultedinchangestothe
generativemodel(i.e,. Bayesiannetwork)thatreflectedan 2.1.Low-FidelityData
absenceofinstrumentationinthelow-fidelity(i.e.,online)
simulation environment. These differences didn’t lead to In the low-fidelity data collection study, baseline driving
significantchangesinthepredictedactiondistributionsbe- conditions (i.e., In-vehicle signage (IVS) technology ab-
MultifidelityRiskEstimation-SubmittedtoICML2017
sentandroadsidesignsavailable)wereincludedasacon- cellular phone that was mounted to the center console of
trol,inordertocomparechangesinrelativesafetyanddriv- thevehiclewithinthedriversfieldofview. Thedescription
ing performance when using the IVS information. This ofeachzonetype,thespeedlimitforthezone. IVSinfor-
low-fidelityconditionwasselectedsinceitreflectsthetech- mationwaspresentedvisuallyonlywithnoaccompanying
nologyavailabletodriversinhigh-fidelity(i.e,real-world) auditoryalertorverbalinformation. Thelow-fidelitydata
settings. Therefore, by comparing a baseline condition collectionmethodsandparticipantsaredescribedindetail
to the two IVS conditions, one in which IVS information inaMnDOTtechnicalreport(Schlicht&Morris,2016).
appears in conjunction with external (i.e., roadside) signs
(IVS+ES)andoneinwhichIVSinformationpresentedin 2.2.High-FidelityData
replacementofexternalsigninformation(IVS-ES),itwas
Thehigh-fidelityspeeddatathatwasprovidedbytheMin-
expected that we would be able to identify any safety and
nesota Department of Transportation (MnDOT) for this
workloadeffectsthatmaybeassociatedwiththeIVStech-
effort. Participants included motorists who were on the
nology.
roadduringMnDOTdatacollection,whichoccurredacross
More specifically, a 2 (Technology Condition: IVS +ES, speed zones during year 2014. We have no reason to be-
IVS -ES) x 2 (IVS Condition: IVS absent (baseline - lievethattheparticipantssampledfromthisdatacollection
only roadside signage), IVS present) mixed-subjects fac- effortdifferedsignificantlyfromthosesampledduringour
torial design was utilized, where subjects were randomly low-fidelitydatacollection(Schlicht&Morris,2016).
assigned to a technology condition (between-subjects fac-
Theroadwaysusedduringthishigh-fidelitydatacollection
tor),butparticipatedineachIVScondition(within-subjects
werethosewhichhaveatleast1000AnnualAverageDaily
factor). Noticethatthisdesignallowsustocheckforboth
Traffic(AADT).Thespeedof100vehiclesweremeasured
maineffectsandinteractionsassociatedwiththeIVStech-
in each direction, and an attempt was made to ensure the
nology.
sampling was done on a clear and sunny day during the
MnDOTspeedassessment. Moreover,samplesweretaken
fromanunmarkedcar,andtheyonlysampledvehiclesthat
aredrivingunderfreeflowconditions.
2.3.ComparisonofEmpiricalSpeedData
Acomparisonbetweenourhigh-fidelityspeeddataandthe
observedlow-fidelitysimulationdataisdepictedinFigure
3. Kernel density estimation (with kernel width of 1) was
performed on both sets of speed data in order to estimate
p(s|sp,c=baselinetechnology),whichistheprobability
oftheobservedspeed(s),giventhepostedspeed(sp)and
(external-signsonly)technologycondition(c).
Figure2.Low-fidelitydatacollectionenvironmentandsimulated
Figure3a-dshowsthehigh-fidelityspeeddistributionsfor
IVStechnology
each of the speed zones that were sampled in our high-
fidelity data collection (40, 50, 55, 60, and 65mph). The
Thisstudywasconductedinapartialmotion-basedriving
low-fidelitydatadepictedareonlyforthesimulationbase-
simulator manufactured by Realtime Technologies (RTI).
line conditions that overlap with our high-fidelity speed
Thesimulatorconsistedofa2002SaturnSC2fullvehicle
zones(Figure3a-c).
cab featuring realistic control operation and instrumenta-
tion including power-assist for the brakes and force feed- Itisapparentthatthereisagreatdegreeofoverlapbetween
backforthesteering. Hapticfeedbackwasprovidedbycar our observed low- and high-fidelity speed distributions
bodyvibrationandathree-axiselectricmotionsystempro- across similar posted speed zones and technology condi-
ducing roll, pitch and yaw motion within a limited range tions. Thedegreeofoverlapcanbequalitativelymeasured
of movement. The auditory feedback was provided by a using K-L Divergence (K(P||Q) = (cid:80) log (p /q )p ),
i 2 i i i
3D surround sound system. The driving environment was where the P = p (s | sp,c = baseline technology)
HiFi
projectedtoafive-channel,210-degreeforwardvisualfield distributionisdefinedtobethetruedistribution(i.e.,high-
screen(2.5arc-minutesperpixel)withrearandsidemirror fidelity speed distribution), that we approximate by using
viewsprovidedbyarearscreenandvehicle-mountedLCD the Q = p (s | sp,c = baseline technology) distribu-
LoFi
panels,respectively. (Figure2). tion(i.e.,low-fidelityspeeddistribution).
IVS information was displayed to drivers on an Android InInformationTheory,K-LDivergenceisameasureofin-
MultifidelityRiskEstimation-SubmittedtoICML2017
2.4.ImpactofIVSTechnologyonLow-FidelitySpeed
0.15 0.15
AEpffpicrioexnicmya: t6io4n% HLoiFFii:: RSeimalu-Wlatoiorlnd AEpffpicrioexnicmya: t9io5n% HLoiFFii:: RSeimalu-Wlatoiorlnd
Probability0.00.51 Probability0.00.51 de113400 BIIVVaSSs e+-ElEinSSe
(a) (b) epS120
00 20Observ4e0d Spe6e0d 80 100 00 20 Obs4e0rved S6p0eed 80 100 detsoP110
tn100
0.15 AEpffpicrioexnicmya: t8io3n% HLoiFFii:: RSeimalu-Wlatoiorlnd 0.15 HiFi: Real-World (60 &65) ecreP n 90
Probability0.00.51 Probability0.00.51 aideM 7800
60
IVS +ES IVS -ES
(c) (d) Condition
0 0
0 20 40 60 80 100 0 20 40 60 80 100
Observed Speed Observed Speed
Figure4.Median Low-fidelity speeding behavior across IVS
technologyconditions
Figure3.Comparison of low- and high-fidelity empirical speed
distributions
Figure4showsthemedianpercentpostedspeedaveraged
acrossallparticipantsiandzoneszduringlow-fidelitydata
collection. Speed was compared for the conditions when
formation gain due to the use of an approximation to the IVS information was presented in conjunction with road-
true distribution, rather than the distribution itself (Cover side signs (IVS +ES) and conditions in which IVS infor-
& Thomas, 2006). Since we know the true (high-fidelity) mation was presented in the absence of roadside sinage
distribution(P)oftheobservedspeeds,itispossibletode- (IVS -ES). We found that percent posted speed was rela-
scribe the distribution with an average description length tivelyconsistentacrosstechnologyconditionsforthebase-
(cid:80)
equaltotheShannonEntropy(H(P)=− p log (p )). line (M = 106.79,SE = 1.10) and IVS +ES condition
i i 2 i
However, if we instead used the low-fidelity distribution (M = 106.44,SE = 1.04). However, there were dif-
(Q) to approximate P, we would need H(P)+K(P||Q) ferencesinmedianpercentpostedspeedacrosstechnology
bits, on average, to describe high-fidelity speed. In other conditionsforthebaseline(M = 106.42,SE = .94)and
words, we need I(Q) = K(P||Q)/H(P) × 100% more IVS-ES(M =123.90,SE =1.31)conditions.
bitsofinformationifweweretousethelow-fidelityspeed
To explore if these differences were significant, a mixed-
data (Q) as an approximation to the high-fidelity distribu-
factorialANOVAwasperformedonthedata. Themixed-
tion(P). Werefertothisquantity(E(Q)=100%−I(Q))
factorial ANOVA found the observed differences in speed
theapproximationefficiencyanditisprovidedfortherele-
to be significant, as there were significant main-effects of
vantconditionsinFigure3a-c.
both IVS condition (F(1,636) = 141.41,p < .01) and
As the E(Q) metric suggests, the low-fidelity data pro- technology conditions (F(1,636) = 25.66,p < .01).
videareasonablyefficientapproximationtothereal-world Moreover,therewasasignificantinteractionbetweentech-
distributions (mean E(Q) = 81% efficiency). This re- nologyandIVSconditions(F(1,636)=121.04,p<.01),
sult agrees with previous theoretical work (Schlicht et al., where those in the IVS -ES group displayed significantly
2012) that demonstrated small differences between low- greater speeds over posted values than those in the other
andhigh-fidelityenvironmentsdonotadverselyimpactthe conditions. Clearly,thisinteractionwasdrivingthesignifi-
ability of the low-fidelity data to be used to predict high- cantmain-effectsthatwereobserved.
fidelityperformance.
AlthoughitisobviousthatincreasedspeedintheIVS-ES
Thisisimportantifwewishtotrainourmodelusinglow- conditionwillresultinincreasedpropertydamageandin-
fidelity data without quantitative adjustment. If there was juryseverityintheeventofacrash,itisunclearaboutthe
lowerapproximationefficiencybetweenthelow-andhigh- extenttowhichthisistrue. Afterall,ifin-vehiclesignin-
fidelity speed data, then quantitative methods would have formationweretoreplaceexternalsigns,itwouldpresum-
tobeleveragedinordertocombinethedatainameaning- ably save money on infrastructure costs, so it is desirable
ful manner; previous theoretical studies investigated such to understand the balance of these two factors in order to
methods (Schlicht et al., 2012) and when they should be makeaninformeddecisionabouttherelativeutilityofthe
leveraged. IVStechnology.
MultifidelityRiskEstimation-SubmittedtoICML2017
Inordertoprovideaproof-of-conceptofhowsuchanesti-
matecouldbeestablished,wefocusedonhowtheobserved (cid:88) (cid:88)
E(V (o))= L(o ) p(o |a ,s )p(s |sp,i,c)p(sp)p(i)
increases in speed will impact the expected lives lost, in c j j t i i
the event of different types of crashes. The next section j i∈I
(cid:88)
will overview the predictive model and risk analysis done = L(o )p(o |a ,s)p(s|sp,c)p(sp)
j j t
tothatend. j
(1)
3.MultifidelityRiskEstimation whereE(V (o))istheexpected-valueassociatedwithIVS
c
conditioncacrossoutcomeso. L(o )isthelossassociated
j
In order to estimate the risk involved with a system, per-
withrealizingoutcomej,andp (o |s,a )istheprobabil-
c j t
formance needs to be evaluated across several conditions.
ityofrealizingoutcomej, giventhespeedsandaccident
Itisintractabletorunhuman-in-the-loopexperimentsover type t. The conditional probability p(s | sp,c) represents
alltypesofpeopleandsituations. Moreover,itisimpossi-
theprobabilityofobservingspeedsgiventhepostedspeed
ble to leverage high-fidelity data when evaluating the risk sp and IVS condition c. This conditional probability was
associatedwithasystemthathasyettobedeployed.There-
derived by marginalizing over individual participant fac-
fore, it is desirable to develop a model that is able to pre-
tors, (I), which also impact the rates of observed speed.
dict human performance across the technological and en- Finally,p(sp)isthemarginalprobabilityofpostedspeeds
vironmentalconditionsofinterest. Themodelcanbeused
acrossaregion(temporalorgeographical)ofinterest.
in Monte-Carlo simulation to evaluate the risk associated
with deploying different types of IVS technology. This Inthegeneralcase,L(oj)couldquantifypropertydamage,
section will describe the model and assumptions used for injuryseverityandfatalityrates. However,forthepurpose
oureffortstothisend. Inthecontextofthecurrentstudy, ofthecurrentstudy,wewillutilizealoss-functionthatfo-
cuses on fatality rates. More specifically, our loss func-
tion simply provides a unit reward for a positive outcome
L(o = 1) = +1, andanegativeunitpenaltyforafatality
L(o=0)=−1.
Notice that in order to reliably estimate p(s | sp,c), it re-
quiresthatwequantifythedistributionofobservedspeeds
i sp a
t in the real-world across various speed zones (sp), operat-
ing under different IVS technology conditions c. This is
intractable, as we can only use high-fidelity traffic data to
estimate this distribution for the baseline condition p(s |
c s o
i sp,c = baseline technology); the IVS conditions do not
occur outside our low-fidelity simulation, so we need to
developamodelthatcanhelpuspredictp(s|sp,c)forthe
IVS+ESandIVS-ESconditions.
L
Candidate predictive models were proposed in the order
of increasing complexity (i.e., in the number of features
andmodelparameters). Itwasdiscoveredthataquadratic
Figure5. BayesiannetworkforIVSriskestimation
model seemed to be the best balance between model sim-
plicity and predictive performance. We converged on a
verysimplenonlinearmodelthatusesonlytwofeatures:
we would like to estimate the fatality risk associated with 1. Speed in the baseline condition for each zone (i.e.,
different IVS technology conditions (c). Figure 5 shows postedspeedzone,z),averagedacrossparticipantssb.
z
theBayesiannetworkusedfortheIVSMonte-Carlosimu- Essentially,thisfeatureencodesaveragespeedingbe-
lation. Shaded circles represent observable random vari- haviorforaparticularpostedspeedzonewhennoIVS
ables, white circles represent unobservable random vari- ispresent.Noticethatthisfeaturecanbeestimatedvia
ables, squares represent the loss function associated with high-fidelityorlow-fidelitydata.
realizing the possible outcomes (o), and arrows represent
2. IVSconditionindicatorvariableencodingtheabsence
causalrelationshipsbetweenvariables.
orpresenceofexternalsignsδ ∈0,1,respectively.
c
Formally,thisdirected-acyclicgraph(DAG)representsthe
followingprobabilisticrelationship: Specifically,themodeltakes-onthefollowingform:
MultifidelityRiskEstimation-SubmittedtoICML2017
Inordertoestimatep(sc |s ,c=baselinetechnology),we
z z
sc =w +w ∗sb +w ∗δ +w ∗(sb)2 (2) useddatafroma2014MnDOTATRreport.Speeddatawas
z 0 1 z 2 c 3 z
providedforfourdifferentspeedzones(40,50,55and60
Figure 6 shows the model performance on training data mph),andthefrequencyoftheobservedspeedsforagiven
across baseline speeds for each IVS condition. The solid zonewerebinnedusingbinsizesof5mph. Sinceweneed
coloredlineshowsthemeanpredictionacrosseachofthe to estimate the average baseline speed to use as a feature
10modelweightswestimatedfromthek-foldcrossvalida- forourpredictivemodel,wecomputedtheweightedaver-
tionprocedure,andtheshadedregionsrepresent±1SEM. agespeedobservedineachzoneonceanhour,andthere-
sultingdistributionofthosehourlyspeedswasusedtoesti-
matep(sc |s ,c=baselinetechnology). Usingthishigh-
z z
fidelityspeeddata,weperformedkerneldensityestimation
65 on the average baseline speed distributions for each zone
Predicted IVS -ES
60 Predicted IVS +ES (z). Then,averagespeedsweresampledfromthebaseline
Actual IVS -ES
Actual IVS +ES distribution(N=10,000),andthecorrespondingspeedsfor
55
)h thedifferentIVStechnologyconditionswerepredictedby
p
m
( d50 usingthebaselinesamplesasinputsintoEquation2. Now
e
epS45 that we have predicted observed speeds scz for each IVS
SV condition (c) and posted speed zone (z), we need to esti-
I40
mate p(sc | s ,c). This was also accomplished by using
z z
35 kerneldensityestimationwithakernelwidthoftwo. Fig-
ure 8 shows the results of estimating distributions associ-
30
30 35 40 45 50 55 60
ated with each IVS technology condition , marginalizing
Baseline Speed(mph)
overallpostedspeedzonesp(sc |c)=(cid:80) p(sc |s ,c).
z z z
Figure6. Trainingperformanceofmodel
Figure 7 shows the predictive performance of the model.
0.06
Predictive performance was evaluated using k-fold cross-
Baseline
validation,wherethedatasetispartitionedintodatathatis IVS +ES
0.05 IVS -ES
usedtoestimatemodelparameters(i.e.,trainingdata)and
data that is used to evaluate predictive performance (i.e., 0.04
testdata). Thispartitioningprocedureisperformedk =10 ytilib
a0.03
b
times across the data and the results of the models ability o
rP
topredictthetestdataisshownonFigure7. Ifthemodel 0.02
resultedinperfectprediction,allthedatawouldfallonthe
0.01
dashed-line. Asthefigureshows, thequadraticmodelde-
fined in Equation 2 is able to predict speed in each IVS 0
30 40 50 60 70 80
conditionfrombaselinedata,withamedianpredictioner- Observed Speed (mph)
rorof±2.2mph.
Figure8. Samplingdistributionsusedinriskestimation
The distribution p (o = 1 | a = t,s) needs to be esti-
c j t
70 matedfromhigh-fidelitydata,aswedidnotexperienceany
65 Median Prediction Error = 2.15 crashesinlow-fidelitysimulation. Thiswasaccomplished
)h60 byleveraginghigh-fidelitytrafficdatafromaninternational
pm source.1 Probitregressionwasutilizedtorecreatethefatal-
( d55
ee itycurvesforeachofthethreecrashtypes(i.e.,pedestrian,
pS d50 side-impact,front-impact).Morespecifically,theprobabil-
e
tcid45 ity of fatality for each speed and crash type was given by
e
rP40 p (o = 0 | a = t,s), and is depicted in Figure 9. The
c t
35 probability of surviving a crash pc(o = 1 | at = t,s) is
simply1−p (o=0|a =t,s).Althoughthehigh-fidelity
30 c t
30 35 40 45 50 55 60 65 70 dataarefromaninternationalsource,theresultsshouldbe
Actual Speed (mph)
1Data adapted from Victorian Government Online Report,
Figure7. Predictiveperformanceofmodel Figure2C
MultifidelityRiskEstimation-SubmittedtoICML2017
valid across a wide range of situations, as the data should expected-value can be intuited as the expected number of
notbestronglycorrelatedwithgeographicalfactors. How- lives lost in the event of an unmitigated crash of type t.
ever,themarginalprobabilityofcrashtypep(a )mayvary Expected-value, in this case, ranges between -1 and +1.
t
acrossregions,butthiswillnotimpactourresults,asweare Thegreatertheexpected-value,thegreaterthesafetyofthe
computingriskseparatelyacrosscrashtypes,asdescribed system. In this respect, we are able to evaluate the safety
below. associated the IVS technology, and the goal of designing
an IVS system should be to maximize the EV associated
withthetechnology,whichcorrespondstominimizingthe
expectedrisk.
1
ytila0.8
ta
F fo0.6 0
y BL Pedestrians
tilib0.4 -0.2 BL Side
ab e BL Front
o Pedestrians u
rP0.2 Side Impact al-0.4
Front Impact V
0 d
e
0 20 40 60 80 100 ct-0.6
Speed (mph) e
p
x
E-0.8
Figure9. Probabilityoffatalityacrossaccidenttypes
-1
Finally, in order to simplify our expected value calcula-
tions,weonlycomputetheexpected-valueassociatedwith IVS +ES IVS -ES
eachIVSconditionintheeventofancrash. Moreover,we
computethisseparatelyforeachcrashtype,whichsimpli- Technology
fiesEquation1to:
(cid:88) Figure10.Average risk across technology conditions and acci-
E(Vt(o))= L(o )p(o |a =t,s)p(s|sp,c)p(sp)
c j j t denttypes
j
(3)
TocomparerelativesafetyoftheIVStechnology,wecom-
Equation3iswhatwasutilizedtoevaluateriskfortheIVS puted the EV associated with the baseline condition (Fig-
conditions in this study. Leveraging the loss function and ure 10, horizontal colored lines). It was found that EV in
probabilityestimatesdescribedabove,wewereabletoper- the baseline condition for an crash involving a pedestrian
formMonte-Carlosimulation(Algorithm1)toproducethe was the least safe (M = -.99, SE = .01), and EV associ-
IVSriskestimatesfoundinFigure10. ated with a side- and (M = -.92, SE = .01) front-impact
was also negative (M = -.37, SE = ,01). The negative es-
Algorithm1IVSMonte-CarloRiskSimulation timated risk is what would be expected from the fatality
1: forn∈Nsimulationsdo curvesdepictedinFigure9,andthespeeddistributionsthat
2: sp ←sample∼p(sp) wereprovidedbyMnDOTandusedforthisrisksimulation
n
3: forc∈C do (Figure 8). For example, the near negative-one EV asso-
4: sc ←sample∼p(s|spn,c) ciatedwithcrashesinvolvingapedestrianresultsfromthe
5: fort∈T do fact that almost all the observed speeds for which we had
6: ptc ←p(oj =0|at =t,sc) high-fidelitydataaregreaterthanthespeedassociatedwith
7: evct,n ←(ptc×L(o=0))+((1−ptc)×L(o= p(oj =0|at =t,sc)=1.
1))
Expected-value for the IVS +ES condition demonstrates
8: endfor
that the technology actually improved safety relative to
9: endfor
baselineconditionsinthecaseofancrashinvolvingafront
10: endfor
impact(M=-0.21,SE=.01).However,forcrashesinvolv-
11: E(Vt(o))= 1 ×(cid:80) evt
c N n c,n ingside-impact(M=-.94,SE=.01)andpedestrians(M=-
.99,SE=.01),EVwascomparabletolevelsinthebaseline
Figure 10 shows the average EV associated with differ- condition, suggesting that the IVS technology minimally
entcrashtypesacrossIVSconditions. Inthisformulation, impactsthesafetyassociatedwiththosecrashtypes.
MultifidelityRiskEstimation-SubmittedtoICML2017
Finally, expected-value for the IVS condition without ex- ment. This allows for candidate technologies to be evalu-
ternalsigns(-ES)demonstratesthatthetechnologysignifi- ated at the stage of conception, and enables a mechanism
cantlydecreasedsafetyrelativetobaselineconditions.This foronlythesafestandmosteffectivetechnologytobede-
was true across crashes involving pedestrians (M = -1.00, velopedandreleased.
SE=.01),side-(M=-.99,SE=.01)andfront-impact(M
=-.65,SE=.02). Clearly,theincreasesinspeedobserved
Acknowledgements
(Figure4)intheIVS-ESconditionadverselyimpactscrash
safetyexpectation. ThisanalysiswasfundedMinnesotaLocalRoadResearch
Board and performed under MnDOT Contract Number
4.Conclusions 99008,WorkOrderNumber145. AspecialthankstoVic-
torLund,projectTechnicalLiasonandTrafficEngineerof
This study proposed a multifidelity method to quantita- St. Louis County, MN. The authors would like to thank
tivelyevaluatetheriskassociatedwithtransportationtech- JanetCreaser,MichaelManser,andPeterEasterlundofthe
nology, prior to deployment. As proof-of-concept, we es- HumanFIRST Program, in addition to Alec Gorjestani of
timatedtheriskassociatedwithin-vehiclesign(IVS)tech- theIntelligentVehiclesLaboratoryfortheirassistancedur-
nologyandfoundthatrelyingonIVSinformationaloneled ingtheexperimentaldesignandlow-fidelitydatacollection
to a significant increase in expected fatalities for crashes portion of this project. Moreover, we would like to thank
involvingfront-andside-impact,relativetostatusquocon- Derek Leuer, Nathan Drews and Eric DeVoe of MnDOT
ditions (Figure 10). The increased risk was the result of forprovidingsomeofthehigh-fidelitytrafficdatausedfor
increasesinspeedunderIVStechnologyconditionswhere ourriskanalysis.
externalsignswereabsent(Figure4).
However,theanalysisalsodiscoveredthatpresentingIVS References
information led to decreased risk when paired with exter-
Agamennoni, G., Nieto, J., and Nebot, E. Estimation of
nalsigns(Figure10),relativetoconditionswhereexternal
multivehicle dynamics by considering contextual infor-
roadside signs were only present. More specifically, risk
mation.IEEETransactionsonRobotics,28(4):855–870,
analysissuggeststhattherearefewerexpectedfatalitiesin-
2012.
volvingfront-impactcollisions. Interestingly, inthiscase,
thereductioninriskwasnottheresultfromdecreasedaver-
Chryssanthacopoulos, J.P. and Kochenderfer, M.J. Col-
agespeeds(Figure4),butratherfromthepropertiesofthe
lision avoidance system optimization with probabilistic
distribution of speeds predicted under this IVS condition
pilotresponsemodels.InProceedingsofAmericanCon-
(Figure 8). Therefore, technology evaluation that exclu-
trol Conference (ACC), pp. 1207–1216, San Francisco,
sivelyreliedonbehavioralspeedaveragesfromlow-fidelity
CA,2011.
simulationwouldhavebeenremiss.
Cover, T.M. and Thomas, J.A. Elements of Information
This study presented a novel multifidelity method for es-
Theory. JohnWileyandSons,2ndedition,2006.
timatingtheriskassociatedwithtransportationtechnology
prior to deployment. Future extensions could include the
Cutler, M., Walsh, T.J., and How, J.P. Real-world rein-
addition of monetary concerns; both direct (e.g., cost of
forcement learning via multifidelity simulators. IEEE
collisionrepair)andcomprehensive(e.g.,wageslost)costs
TransactionsonRobotics,31:655–671,2015.
couldbeincorporatedintothelossfunction. Bydoingthis,
risk would better reflect concerns of individuals who are
Gietelink,O.,Ploeg,J.,Schutter,B.De,andVerhegen,M.
responsibleforevaluatingtransportationtechnology.
Developmentofadvanceddriverassistancesystemswith
Futureworkcouldincludeupdatingthefatalityprobability vehicle hardware-in-the-loop simulations. Vehicle Sys-
curveswithdatafromdomesticsources. Moreover,thefa- temDynamics,44(7):569–590,2006.
talityprobabilitycurvescouldbeconditionedonotherfac-
torsthatareknowntochangewithmodificationsintrans- Gindele,T.,Brechtel,S.,andDillmann,R. Learningcon-
portationtechnology. Ideally,ourprobabilitycurveswould textsensitivebehaviormodelsfromobservationsforpre-
also include a model for how variability in speed impacts dicting traffic situations. In Langley, Pat (ed.), in IEEE
factorsthatarebeingconsideredbythelossfunction(e.g., International Conference on Intelligent Transportation
fatalities,cost). Systems(ITSC),pp.17641771,2013.
Overall, these results provide a proof-of-concept for how
Gruening, J., Bernard, J., Clover, C., and Hoffmeister, K.
multifidelity models can be leveraged to estimate the risk
Drivingsimulation. InSocietyofAutomotiveEngineers
associatedwithtransportationtechnology,priortodeploy-
SAEPaperNo.980223,1998.
MultifidelityRiskEstimation-SubmittedtoICML2017
Lee,H.C.,Lee,A.H.,Camerson,D.,andLi-Tsang,C. Us-
ing a driving simulator to identify older drivers at in-
flated risk of motor vehicle crashes. Journal of Safety
Research,34:453–459,2003.
Lefevre,S.,Sun,C.,Bajcsy,R.,andLaugier,C. Compari-
sonofparametricandnon-parametricapproachesforve-
hiclespeedprediction. InAmericanControlConference
(ACC),pp.34943499,2014.
Maki, E., Weinert, A., andKochenderfer, M.J. Efficiently
estimating ambient near mid-air collision risk for un-
manned aircraft. In in AIAA Aviation Technology, In-
tegration, and Operations Conference, Fort Worth, TX,
2010.
Morton,J.,Wheeler,T.A.,andKochenderfer,M.J. Analy-
sisofrecurrentneuralnetworksforprobabilisticmodel-
ingofdriverbehavior. InIEEETransactionsonIntelli-
gentTransportationSystems,2016.
Mueller,E.andKochenderfer,M.J.Simulationcomparison
of collision avoidance algorithms for small multi-rotor
aircraft. In in AIAA Modeling and Simulation Confer-
ence,2016.
Robinson, T., Willcox, K., Eldred, M., and Haimes, R.
Multifidelity optimization for variable-complexity de-
sign. Inin11thAIAA/ISSMOMultidisciplinaryAnalysis
andOptimizationConference,2006.
Schlicht, E.J. and Morris, N.L. Risk evaluation of in-
vehicle sign information. Technical report, Minnesota
Department of Transportation (MnDOT) Technical Re-
port,2016-18,St.Paul,MN,2016.
Schlicht,E.J.,Lee,R.,Kochenderfer,M.J.,Wolpert,D.W.,
andTracy,B. Predictingthebehaviorofinteractinghu-
mans by fusing data from multiple sources. In 28th
ConferenceonUncertaintyinArtificialIntelligence,UAI
2012,pp.756–764,CatalinaIsland,CA,2012.
Shechtman, O., Classen, S., Awadzi, K., and Mann, W.
Comparison of driving errors between on-the-road and
simulateddrivingassessment: Avalidationstudy. Traf-
ficInjuryPrevention,10:379–385,2009.
Wheeler, T.A., Robbel, P., and Kochenderfer, M.J. Anal-
ysis of microscopic behavior models for probabilis-
tic modeling of driver behavior. In in IEEE Interna-
tionalConferenceonIntelligentTransportationSystems
(ITSC),RiodeJaneiro,Brazil,2016.
Yang, Q. and Koutsopoulos, H.N. A microscopic traffic
simulatorforevaluationofdynamictrafficmanagement
systems. TrafficInjuryPrevention,4:113–129,1996.