Predicting Citywide Crowd Flows Using Deep Spatio-Temporal Residual Networks

5.9 MB·

by  Junbo Zhang

Predicting Citywide Crowd Flows Using
(cid:73)
Deep Spatio-Temporal Residual Networks
JunboZhanga,YuZhenga,b,c,d,∗,DekangQib,1,RuiyuanLic,1,XiuwenYib,1,TianruiLib
aMicrosoftResearch,Beijing,China
bSchoolofInformationScienceandTechnology,SouthwestJiaotongUniversity,Chengdu610031,China
cSchoolofComputerScienceandTechnology,XidianUniversity,China
dShenzhenInstitutesofAdvancedTechnology,ChineseAcademyofSciences
Abstract
7
1 Forecastingtheflowofcrowdsisofgreatimportancetotrafficmanagementandpublicsafety,andverychallenging
0 asitisaffectedbymanycomplexfactors,includingspatialdependencies(nearbyanddistant),temporaldependencies
2
(closeness, period, trend), and external conditions (e.g. weather and events). We propose a deep-learning-based

n
approach, called ST-ResNet, to collectively forecast two types of crowd flows (i.e. inflow and outflow) in each and
a
everyregionofacity. Wedesignanend-to-endstructureofST-ResNetbasedonuniquepropertiesofspatio-temporal
J
data. Morespecifically,weemploytheresidualneuralnetworkframeworktomodelthetemporalcloseness,period,
0
1 and trend properties of crowd traffic. For each property, we design a branch of residual convolutional units, each
of which models the spatial properties of crowd traffic. ST-ResNet learns to dynamically aggregate the output of

] thethreeresidualneuralnetworksbasedondata, assigningdifferentweightstodifferentbranchesandregions. The
I
A aggregationisfurthercombinedwithexternalfactors,suchasweatheranddayoftheweek,topredictthefinaltraffic
. ofcrowdsineachandeveryregion. Wehavedevelopedareal-timesystembasedonMicrosoftAzureCloud, called
s
UrbanFlow,providingthecrowdflowmonitoringandforecastinginGuiyangCityofChina. Inaddition,wepresent
c
[ an extensive experimental evaluation using two types of crowd flows in Beijing and New York City (NYC), where

ST-ResNetoutperformsninewell-knownbaselines.
1
v
Keywords: ConvolutionalNeuralNetworks,Spatio-temporalData,ResidualLearning,CrowdFlows,Cloud
3
4
5
2
0
1. Introduction
.
1
0 Predictingcrowdflowsinacityisofgreatimportancetotrafficmanagement,riskassessment,andpublicsafety
7 [2]. Forinstance,massivecrowdsofpeoplestreamedintoastripregionatthe2015NewYear’sEvecelebrationsin
1
Shanghai,resultinginacatastrophicstampedethatkilled36people. Inmid-Julyof2016,hundredsof“PokemonGo”
:
v playersranthroughNewYorkCity’sCentralParkinhopesofcatchingaparticularlyraredigitalmonster,leadingtoa
i dangerousstampedethere. Ifonecanpredictthecrowdflowinaregion,suchtragediescanbemitigatedorprevented
X
byutilizingemergencymechanisms,suchasconductingtrafficcontrol,sendingoutwarnings,orevacuatingpeople,
r
a inadvance.
Inthispaper,wepredicttwotypesofcrowdflows[3]: inflowandoutflow,asshowninFigure1(a). Inflowisthe
totaltrafficofcrowdsenteringaregionfromotherplacesduringagiventimeinterval.Outflowdenotesthetotaltraffic
of crowds leaving a region for other places during a given time interval. Both types of flows track the transition of
(cid:73)Thispaperisanexpandedversionof[1],whichhasbeenacceptedforpresentationatthe31st AAAIConferenceonArtificialIntelligence
(AAAI-17).
∗Correspondingauthor
Emailaddresses:[email protected](JunboZhang),[email protected](YuZheng),[email protected]
(DekangQi),[email protected](RuiyuanLi),[email protected](XiuwenYi),[email protected](TianruiLi)
1Theresearchwasdonewhenthethird,fourthandfifthauthorswereinternsatMicrosoftResearch.
PreprintsubmittedtoElsevier January11,2017
crowdsbetweenregions.Knowingthemisverybeneficialforriskassessmentandtrafficmanagement.Inflow/outflow
canbemeasuredbythenumberofpedestrians,thenumberofcarsdrivennearbyroads,thenumberofpeopletraveling
on public transportation systems (e.g., metro, bus), or all of them together if data is available. Figure 1(b) presents
anillustration. Wecanusemobilephonesignalstomeasurethenumberofpedestrians,showingthattheinflowand
outflowofr are(3,1), respectively. Similarly, usingtheGPStrajectoriesofvehicles, twotypesofflowsare(0,3),
2
respectively. Therefore,thetotalinflowandoutflowofr are(3,4),respectively. Apparently,predictingcrowdflows
2
canbeviewedasakindofspatio-temporalpredictionproblem[2].
(a) Inflowandoutflow (b) Measurementofflows
Figure1:Crowdflowsinaregion
Deeplearning[4]hasbeenusedsuccessfullyinmanyapplications,andisconsideredtobeoneofthemostcutting-
edgeartificialintelligence(AI)techniques.Exploringthesetechniquesforspatio-temporaldataisofgreatimportance
toaseriesofvariousspatio-temporalapplications,includingurbanplanning,transportation,theenvironment,energy,
social, economy, public safety and security [2]. Although two main types of deep neural networks (DNNs) have
considered a sort of spatial or temporal property: 1) convolutional neural networks (CNNs) for capturing spatial
structures; 2) recurrent neural networks (RNNs) for learning temporal dependencies. It is still very challenging
to apply these existing AI techniques for such spatio-temporal prediction problem because of the following three
complexfactors:
1. Spatialdependencies.
Nearby The inflow of Region r (shown in Figure 1(a)) is affected by outflows of nearby regions (like r ).
2 1
Likewise,theoutflowofr wouldaffectinflowsofotherregions(e.g.,r ). Theinflowofregionr would
2 3 2
affectitsownoutflowaswell.
Distant Theflowscanbeaffectedbythatofdistantregions. Forinstance,peoplewholivesfarawayfromthe
officeareaalwaysgotoworkbymetroorhighway,implyingthattheoutflowsofdistantregionsdirectly
affecttheinflowoftheofficearea.
2. Temporaldependencies.
Closeness Theflowofcrowdsinaregionisaffectedbyrecenttimeintervals,bothnearandfar. Forinstance,a
trafficcongestionoccurringat8:00amwillaffectthatof9:00am.Andthecrowdflowsoftoday’s16thtime
interval2ismoresimilartothatofyesterday’s16thtimeintervalthanthatoftoday’s20thtimeinterval.
Period Trafficconditionsduringmorningrushhoursmaybesimilaronconsecutiveweekdays,repeatingevery
24hours.
Trend Morningrushhoursmaygraduallyhappenlateraswintercomes.Whenthetemperaturegraduallydrops
andthesunriseslaterintheday,peoplegetuplaterandlater.
3. Externalinfluence.Someexternalfactors,suchasweatherconditionsandeventsmaychangetheflowofcrowds
tremendouslyindifferentregionsofacity. Forexample,athunderstormaffectsthetrafficspeedonroadsand
furtherchangestheflowsofregions.
Totackleabovementionedchallenges,wehereexploreDNNsforspatio-temporaldata,andproposeadeepspatio-
temporal residual network (ST-ResNet) to collectively predict inflow and outflow of crowds in every region. Our
contributionsarefive-fold:
2Assumethathalf-hourisatimeinterval,16thtimeintervalmeans7:30am-8:00am.
2
• ST-ResNetemploysconvolution-basedresidualnetworkstomodelbothnearbyanddistantspatialdependencies
betweenanytworegionsinacity,whileensuringthemodel’spredictionaccuracyisnotcomprisedbythedeep
structureoftheneuralnetwork.
• Wesummarizethetemporalpropertiesofcrowdflowsintothreecategories,consistingoftemporalcloseness,
period,andtrend. ST-ResNetusesthreedifferentresidualnetworkstomodeltheseproperties,respectively.
• ST-ResNetdynamicallyaggregatestheoutputofthethreeaforementionednetworks,assigningdifferentweights
todifferentbranchesandregions. Theaggregationisfurthercombinedwithexternalfactors(e.g.,weather).
• WeevaluateourapproachusingBeijingtaxicabs’trajectoriesandmeteorologicaldata,andNYCbiketrajectory
data. Theresultsdemonstratetheadvantagesofourapproachcomparedwith9baselines.
• We develop a real-time crowd flow monitoring & forecasting system using ST-ResNet. And our solution is
basedontheCloudandGPUservers,providingpowerfulandflexiblecomputationalenvironments.
The rest of this paper is organized as follows. In Section 2, we formally describe the crowd flow prediction
problem. Section3overviewsthearchitectureofourproposedsystem. Section4describestheDNN-basedprediction
modelused. WepresenttheevaluationinSection5andsummarizedtherelatedworkinSection6.
The differences between this paper and our earlier work [1] are four aspects. First, we have deployed a cloud-
based system that continuously forecasts the flow of taxicabs in each and every region of Guiyang City in China,
showcasing the capability of ST-ResNet in handling real-world problems. The implementation of the cloud-based
systemisalsointroducedinSection3ofthispaper. Second,weextendST-ResNetfromaone-stepaheadprediction
toamulti-stepaheadprediction,enablingthepredictionofcrowdflowsoverafartherfuturetime(Section4.4).Third,
weconductmorecomprehensiveexperimentsoncrowdflowprediction,showcasingtheeffectivenessandrobustness
of ST-ResNet: i) comparing our method with more advanced baselines (e.g., three different variants of recurrent
neural networks) (Section 5.2); ii) testing more network architectures for ST-ResNet (Section 5.3); iii) adding the
experimentsofmulti-stepaheadprediction(Section5.4);iv)discussingtheperformanceofourmethodchangingover
different cloud resources (Section 5.5). Fourth, we have explored more related works (in Section 6), clarifying the
differencesandconnectionstothe-state-of-the-art. Thishelpsbetterpositionourresearchinthecommunity.
2. Preliminary
Wefirstbrieflyrevisitthecrowdflowpredictionproblem[3]andthenintroducedeepresiduallearning[5].
2.1. FormulationofCrowdFlowPredictionProblem
Definition1(Region[3]). Therearemanydefinitionsofalocationintermsofdifferentgranularitiesandsemantic
meanings. Inthisstudy,wepartitionacityintoan I×J gridmapbasedonthelongitudeandlatitudewhereagrid
denotesaregion,asshowninFigure2(a).
���
���
���
���
��
�
(a) Mapsegmentation (b) Inflowmatrix
Figure2:RegionsinBeijing:(a)Grid-basedmapsegmentation;(b)inflowsineveryregionofBeijing
Definition2(Inflow/outflow[3]). Let P be a collection of trajectories at the tth time interval. For a grid (i, j) that
liesattheithrowandthe jthcolumn,theinflowandoutflowofthecrowdsatthetimeintervaltaredefinedrespectively
as
(cid:88)
xin,i,j = |{k>1|g (cid:60)(i, j)∧g ∈(i, j)}|
t k−1 k
Tr∈P
(cid:88)
xtout,i,j = |{k≥1|gk ∈(i, j)∧gk+1 (cid:60)(i, j)}|
Tr∈P
3
whereTr :g →g →···→g isatrajectoryinP,andg isthegeospatialcoordinate;g ∈(i, j)meansthepoint
1 2 |Tr| k k
g lieswithingrid(i, j),andviceversa;|·|denotesthecardinalityofaset.
k
Atthetthtimeinterval,inflowandoutflowinallI×JregionscanbedenotedasatensorX ∈R2×I×J where(X) =
t t 0,i,j
xin,i,j,(X) = xout,i,j. TheinflowmatrixisshowninFigure2(b).
t t 1,i,j t
Formally,foradynamicalsystemoveraspatialregionrepresentedbyaI×J gridmap,thereare2typesofflows
ineachgridovertime. Thus,theobservationatanytimecanberepresentedbyatensorX∈R2×I×J.
Problem1. Giventhehistoricalobservations{X|t=0,···,n−1},predictX .
t n
2.2. DeepResidualLearning
Deepresiduallearning[6]allowsconvolutionneuralnetworkstohaveasuperdeepstructureof100layers,even
over-1000 layers. And this method has shown state-of-the-art results on multiple challenging recognition tasks, in-
cludingimageclassification,objectdetection,segmentationandlocalization[6].
Formally,aresidualunitwithanidentitymapping[5]isdefinedas:
X(l+1) =X(l)+F(X(l)) (1)
where X(l) and X(l+1) are the input and output of the lth residual unit, respectively; F is a residual function, e.g., a
stackoftwo3×3convolutionlayersin[6]. Thecentralideaoftheresiduallearningistolearntheadditiveresidual
functionF withrespecttoX(l)[5].
3. SystemArchitecture
Figure3presentstheframeworkofoursystem,whichconsistsofthreemajorparts: localGPUservers,andthe
Cloud,andusers(e.g.,websiteandQRCode),resultinginofflineandonlinedataflows,respectively. ThelocalGPU
serversstorehistoricalobservations,suchastaxitrajectories,meteorologicaldata. TheCloudreceivereal-timedata,
including real-time traffic data (e.g. trajectories) within a time interval as well as meteorological data. The users
accesstheinflow/outflowdata,displayingthemonwebsitesorsmartphoneviascanningQRcode.
Figure3:SystemFramework
3.1. TheCloud
ThecloudcontinuouslyreceivesGPStrajectoriesoftaxicabsandcrawlsmeteorologicaldata,andthencachesthem
intoaredis. Avirtualmachine(VM)(orVMs)ontheCloudpullsthesedatafromtheredis,andthencomputescrowd
flowsaccordingtoGPStrajectoriesforeachandeveryregionofacity. Meanwhile,theVMextractsthefeaturesfrom
meteorologicaldata,eventdataandothers. Afterwards,theVMstoresthecrowdflowdataandextractedfeaturesinto
4
thestorage(apartoftheVM).Tosavetheresourceonthecloud(morestoragesneedmoreexpensivepayment),we
onlystorethecrowdflowdataandfeaturesinpasttwodays.Historicaldatacanbemovedtolocalserversperiodically.
We use Azure platform as a service (PaaS). Table 1 details the Azure3 resources for our system as well as the
price4. We employ a VM5, called A2 standard in Azure, that has 2 cores and 3.5 GB memory for forecasting the
crowdflowsinnearfuture. ThewebsiteandwebserviceshareaAppService,giventhepotentialheavyaccessesby
manyusers. Asthehistoricaldataisstoredinlocalservers,a6GBRedisCacheisenoughforcachingthereal-time
trajectoriesinpasthalf-hour,crowdflowdata&extractedfeaturesinpasttwodays,andinferredresults.
Table1:TheAzureresourcesusedforoursystem
AzureService Configuration Price
AppService Standard,4cores,7GBmemory $0.400/hour
VirtualMachine A2standard,2cores,3.5GBmemory $0.120/hour
RedisCache P1premium,6GB $0.555/hour
3.2. LocalGPUServers
Althoughallthejobscanbedoneonthecloud,GPUservicesontheCloudisnotsupportedinsomeareas(e.g.,
China). On the other hand, we need to pay for other cloud services, like storages and I/O bandwidths. Saving
unnecessary cost is vital for a research prototype. In addition, migrating massive data from local servers up to the
cloudistime-consuminggiventhelimitednetworkbandwidth.Forinstance,thehistoricaltrajectoriescanbehundreds
ofGigabytes,evenTerabytes,leadingtoaverylongtimeforcopyingthedatafromlocalserverstothecloud.
Therefore,weemployahybridframeworkthatcombineslocalGPUserverswiththecloud. LocalGPUservers
mainlyhandletheofflinetraining(learning),includingthreetasks:
• Convertingtrajectoriesintoinflow/outflow: wefirstusethemassivehistoricaltrajectoriesandemployacalcu-
lationmoduletogetcrowdflowdata,thenstoretheminlocal.
• Extractingfeaturesfromexternaldata: wefistcollectexternaldata(e.g. weather,holidayevents)fromdifferent
datasourcesandfitthemintoafeatureextractionmoduletogetcontinuousordiscretefeatures,andthenstore
theminlocal.
• Trainingmodel: weuseabovegeneratedcrowdflowsandexternalfeaturestotrainapredictivemodelviaour
proposed ST-ResNet, and then upload the learned model to the cloud. Note that as the dynamic crowd flows
andfeaturesarestoredintheStorage(Azure),wesyncuptheonlinedatatolocalserversbeforeeachtraining
processing. Inthisway,weareagiletotrynewideas(e.g. re-trainthemodel)whilegreatlyreducingexpense
foraresearchprototype.
3.3. UserInterface
Figure4(a)presentsthewebsiteofUrbanFlow[7],whereeachgridonthemapstandsforaregionandthenumber
associated with it denotes its inflow or outflow of crowds. The user can view inflow or outflow via the top-right
button named “InFlow/OutFlow”. The smaller the number is, the crowd flow is sparser. The color of each grid is
determined in accordance with its crowd flows, e.g., “red” means “dense” crowd flow and “green” means “sparse”
crowdflow. Thetop-rightcornerofthewebsiteshowsthebuttonswhichcanswitchbetweendifferenttypesofflows.
A user can select any grid (representing a region) on the website and click it to see the region’s detailed flows, as
showninFigure4(b)whereblue,black,andgreencurvesindicateflowsofyesterday,past,andfuturetimesattoday,
respectively. Thebottomofthewebsiteshowsafewsequentialtimestamps. Theheatmapatacertaintimestampwill
beshowninthewebsitewhenauserclickstheassociatedtimestamp. Intuitively,theusercanwatchthemovie-style
heatmaps(Figure4(c))byclicking“playbutton”atthebottom-leftofFigure4(a). Atpresent,weapplyUrbanFlow
totheareaofGuiyangCity,China6.
3https://azure.microsoft.com
4https://azure.microsoft.com/en-us/pricing
5Toacceleratethecomputation,onealsocanchooseamorepowerfulVM,suchasD4standardofAzurethathas8coresand28GBmemory
with$0.616perhour.
6http://urbanflow.sigkdd.com.cn/
5
Figure4:WebuserinterfaceofUrbanFlow
4. DeepSpatio-TemporalResidualNetworks
Recurrentneuralnetworks(RNNs),likelong-shorttermmemory(LSTM),iscapableoflearninglong-rangetem-
poraldependencies. UsingRNNs,however,tomodeltemporalperiod andtrend,itneedsverylonginputsequences
(e.g., length = 1344)7, which makes the whole training processing non-trivial (see Section 5.2 for empirical eval-
uation). According to the ST domain knowledge, we know that only a few previous keyframes influence the next
keyframe. Therefore, we leverage temporal closeness, period, trend to select keyframes for modeling. Figure 5
presentsthearchitectureofST-ResNet,whichiscomprisedoffourmajorcomponentsmodelingtemporalcloseness,
period,trend,andexternalinfluence,respectively.
Figure5:ST-ResNetarchitecture.Conv:Convolution;ResUnit:ResidualUnit;FC:Fully-connected.
As illustrated in the top-right part of Figure 5, we first turn inflow and outflow throughout a city at each time
interval into a 2-channel image-like matrix respectively, using the approach introduced in Definitions 1 and 2. We
thendividethetimeaxisintothreefragments,denotingrecenttime,near historyanddistanthistory. The2-channel
flowmatricesofintervalsineachtimefragmentarethenfedintothefirstthreecomponentsseparatelytomodelthe
7Assumethathalf-an-hourisatimeinterval,4-weeksequence’slengthisequalto48×7×4=1344.
6
aforementionedthreetemporalproperties: closeness,periodandtrend,respectively. Thefirstthreecomponentsshare
thesamenetworkstructurewithaconvolutionalneuralnetworkfollowedbyaResidualUnitsequence.Suchstructure
capturesthespatialdependencybetweennearbyanddistantregions. Intheexternalcomponent,wemanuallyextract
some features from external datasets, such as weather conditions and events, feeding them into a two-layer fully-
connectedneuralnetwork. TheoutputsofthefirstthreecomponentsarefusedasX basedonparametermatrices,
Res
which assign different weights to the results of different components in different regions. X is further integrated
Res
withtheoutputoftheexternalcomponentX . Finally, theaggregationismappedinto[−1,1]byaTanhfunction,
Ext
whichyieldsafasterconvergencethanthestandardlogisticfunctionintheprocessofback-propagationlearning[8].
4.1. StructuresoftheFirstThreeComponents
Thefirstthreecomponents(i.e. closeness,period,trend)sharethesamenetworkstructure,whichiscomposedof
twosub-components: convolutionandresidualunit,asshowninFigure6.
(a) Convolutions (b) ResidualUnit
Figure6:Convolutionandresidualunit
Convolution. A city usually has a very large size, containing many regions with different distances. Intuitively,
the flow of crowds in nearby regions may affect each other, which can be effectively handled by the convolutional
neuralnetwork(CNN)thathasshownitspowerfulabilitytohierarchicallycapturethespatialstructuralinformation
[9]. Inaddition,subwaysystemsandhighwaysconnecttwolocationswithafardistance,leadingtothedependency
betweendistantregions.Inordertocapturethespatialdependencyofanyregion,weneedtodesignaCNNwithmany
layersbecauseoneconvolutiononlyaccountsforspatialneardependencies,limitedbythesizeoftheirkernels. The
sameproblemalsohasbeenfoundinthevideosequencegeneratingtaskwheretheinputandoutputhavethesame
resolution[10]. Severalmethodshavebeenintroducedtoavoidthelossofresolutionbroughtaboutbysubsampling
whilepreservingdistantdependencies[11]. BeingdifferentfromtheclassicalCNN,wedonotusesubsampling,but
only convolutions [12]. As shown in Figure 6(a), there are three multiple levels of feature maps that are connected
withafewconvolutions. Wefindthatanodeinthehigh-levelfeaturemapdependsonninenodesofthemiddle-level
featuremap,thoseofwhichdependonallnodesinthelower-levelfeaturemap(i.e. input). Itmeansoneconvolution
naturally captures spatial near dependencies, and a stack of convolutions can further capture distant even citywide
dependencies.
The closeness component of Figure 5 adopts a few 2-channel flows matrices of intervals in the recent time to
modeltemporalclosenessdependency. Lettherecentfragmentbe[X ,X ,···,X ],whichisalsoknownas
t−lc t−(lc−1) t−1
theclosenessdependentsequence. Wefirstconcatenatethemalongwiththefirstaxis(i.e. timeinterval)asonetensor
X(c0) ∈R2lc×I×J,whichisfollowedbyaconvolution(i.e. Conv1showninFigure5)as:
(cid:16) (cid:17)
X(1) = f W(1)∗X(0)+b(1) (2)
c c c c
where ∗ denotes the convolution in a convolutional operator; f is an activation function, e.g. the rectifier f(z) :=
max(0,z)[13];W(1),b(1)arethelearnableparametersinthefirstlayer.
c c
Theclassicalconvolutionhassmalleroutputsizethaninputsize, namely, narrowconvolution, asshowninFig-
ure7(a). Assumethattheinputsizeis5×5andthefiltersizeis3×3withstride1,theoutputsizeis3×3ifusing
narrowconvolution. Inourtask,thefinaloutputsizeshouldbesameasthesizeoftheinput(i.e. I×J). Forthisgoal,
weemployaspecialtypeofconvolution,i.e. sameconvolution(seeFigure7(b)),whichallowsafiltertogooutside
theborderofaninput,paddingeachareaoutsidetheborderwithazero.
Residual Unit. It is a well-known fact that very deep convolutional networks compromise training effectiveness
thoughthewell-knownactivationfunction(e.g. ReLU)andregularizationtechniquesareapplied[14,13,15]. Onthe
7
Figure7:Narrowandsametypesofconvolution.Thefilterhassize3×3.
otherhand,westillneedaverydeepnetworktocaptureverylargecitywidedependencies. Foratypicalcrowdflow
data, assumethattheinputsizeis32×32, andthekernelsizeofconvolutionisfixedto3×3, ifwewanttomodel
citywidedependencies(i.e., eachnodeinhigh-levellayerdependsonallnodesoftheinput), itneedsmorethan15
consecutive convolutional layers. To address this issue, we employ residual learning [6] in our model, which have
beendemonstratedtobeveryeffectivefortrainingsuperdeepneuralnetworksofover-1000layers.
InourST-ResNet(seeFigure5),westackLresidualunitsuponConv1asfollows,
X(l+1) =X(l)+F(X(l);θ(l)),l=1,···,L (3)
c c c c
whereF istheresidualfunction(i.e. twocombinationsof“ReLU+Convolution”,seeFigure6(b)),andθ(l) includes
alllearnableparametersinthelth residualunit. WealsoattemptBatchNormalization(BN)[14]thatisaddedbefore
ReLU. On top of the Lth residual unit, we append a convolutional layer (i.e. Conv2 shown in Figure 5). With 2
convolutionsandLresidualunits,theoutputoftheclosenesscomponentofFigure5isX(L+2).
c
Likewise, using the above operations, we can construct the period and trend components of Figure 5. Assume
that there are l time intervals from the period fragment and the period is p. Therefore, the period dependent se-
p
quence is [X ,X ,···,X ]. With the convolutional operation and L residual units like in Eqs. 2 and 3,
t−lp·p t−(lp−1)·p t−p
theoutputoftheperiodcomponentisX(L+2). Meanwhile,theoutputofthetrendcomponentisX(L+2) withtheinput
p q
[X ,X ,···,X ]wherel isthelengthofthetrenddependentsequenceandqisthetrendspan.Notethatp
t−lq·q t−(lq−1)·q t−q q
andqareactuallytwodifferenttypesofperiods. Inthedetailedimplementation, pisequaltoone-daythatdescribes
dailyperiodicity,andqisequaltoone-weekthatrevealstheweeklytrend.
4.2. TheStructureoftheExternalComponent
Traffic flows can be affected by many complex external factors, such as weather and event. Figure 8(a) shows
thatcrowdflowsduringholidays(ChineseSpringFestival)canbesignificantlydifferentfromtheflowsduringnormal
days. Figure 8(b) shows that heavy rain sharply reduces the crowd flows at Office Area compared to the same day
of the latter week. Let E be the feature vector that represents these external factors at predicted time interval t. In
t
ourimplementation,wemainlyconsiderweather,holidayevent,andmetadata(i.e. DayOfWeek,Weekday/Weekend).
ThedetailsareintroducedinTable2.Topredictflowsattimeintervalt,theholidayeventandmetadatacanbedirectly
obtained. However,theweatheratfuturetimeintervaltisunknown. Instead,onecanusetheforecastingweatherat
timeintervaltortheapproximateweatherattimeintervalt−1. Formally,westacktwofully-connectedlayersupon
E, the first layer can be viewed as an embedding layer for each sub-factor followed by an activation. The second
t
layerisusedtomaplowtohighdimensionsthathavethesameshapeasX. Theoutputoftheexternalcomponentof
t
Figure5isdenotedasX withtheparametersθ .
Ext Ext
4.3. Fusion
Inthissection,wediscusshowtofusefourcomponentsofFigure5. Wefirstfusethefirstthreecomponentswith
aparametric-matrix-basedfusionmethod,whichisthenfurthercombinedwiththeexternalcomponent.
Figures9(a)and(d)showtheratiocurvesusingBeijingtrajectorydatapresentedinTable2where x-axisistime
gapbetweentwotimeintervalsandy-axisistheaverageratiovaluebetweenarbitrarytwoinflowsthathavethesame
time gap. The curves from two different regions all show an empirical temporal correlation in time series, namely,
inflowsofrecenttimeintervalsaremorerelevantthanonesofdistanttimeintervals,whichimpliestemporalcloseness.
Thetwocurveshavedifferentshapes,whichdemonstratesthatdifferentregionsmayhavedifferentcharacteristicsof
8
400 400
Normal
Holiday
300 300 Thunderstom
w w
nflo200 nflo200
I I
100 100
0 Mon Tue Wed Thu Fri Sat Sun 0 Sat Sun Mon
Time Time
(a) Feb8-14(red),Feb15-21(green),2016 (b) Aug10-12(red),Aug17-19(green),2013
Figure8:EffectsofholidaysandweatherinOfficeAreaofBeijing(theregionisshowninFigure2(a)).
closeness. Figures9(b)and(e)depictinflowsatalltimeintervalsof7days. Wecanseetheobviousdailyperiodicity
in both regions. In Office Area, the peak values on weekdays are much higher than ones on weekends. Residential
Areahassimilarpeakvaluesforbothweekdaysandweekends. Figures9(c)and(f)describeinflowsatacertaintime
interval (9:00pm-9:30pm) of Tuesday from March 2015 and June 2015. As time goes by, the inflow progressively
decreases in Office Area, and increases in Residential Area. It shows the different trends in different regions. In
summary,inflowsoftworegionsareallaffectedbycloseness,period,andtrend,butthedegreesofinfluencemaybe
verydifferent. Wealsofindthesamepropertiesinotherregionsaswellastheiroutflows.
3 400
300
Ratio12 Inflow120000 Inflow230000
0 0 100
0 5 10 15 20 25 MonTueWedThu Fri Sat Sun Mar 3 Mar 31Apr 28May 26Jun 23
Time Gap (half hour)
(a) Closeness of Office Area (b) Period of Office Area (c) Trend of Office Area
3 200
300
Ratio12 Inflow120000 Inflow110500
0 0 50
0 5 10 15 20 25 MonTueWedThu Fri Sat Sun Mar 3 Mar 31Apr 28May 26Jun 23
Time Gap (half hour)
(d) Closeness of Residential Area (e) Period of Residential Area (f) Trend of Residential Area
Figure9:Temporaldependencies(OfficeAreaandResidentialAreaareshowninFigure2(a)).
Aboveall,thedifferentregionsareallaffectedbycloseness,periodandtrend,butthedegreesofinfluencemaybe
different. Inspiredbytheseobservations,weproposeaparametric-matrix-basedfusionmethod.
Parametric-matrix-based fusion. We fuse the first three components (i.e. closeness, period, trend) of Figure 5 as
follows
X =W ◦X(L+2)+W ◦X(L+2)+W ◦X(L+2) (4)
Res c c p p q q
where◦isHadamardproduct(i.e. element-wisemultiplication), W , W andW arethelearnableparametersthat
c p q
adjustthedegreesaffectedbycloseness,periodandtrend,respectively.
Fusing the external component. We here directly merge the output of the first three components with that of the
external component, as shown in Figure 5. Finally, the predicted value at the tth time interval, denoted by X(cid:98), is
t
definedas
X(cid:98) =tanh(X +X ) (5)
t Res Ext
wheretanhisahyperbolictangentthatensurestheoutputvaluesarebetween-1and1.
9
OurST-ResNetcanbetrainedtopredictX fromthreesequencesofflowmatricesandexternalfactorfeaturesby
t
minimizingmeansquarederrorbetweenthepredictedflowmatrixandthetrueflowmatrix:
L(θ)=(cid:107)X −X(cid:98)(cid:107)2 (6)
t t 2
whereθarealllearnableparametersintheST-ResNet.
4.4. AlgorithmsandOptimization
Algorithm 1 outlines the ST-ResNet training process. We first construct the training instances from the original
sequencedata(lines1-6). Then,ST-ResNetistrainedviabackpropagationandAdam[16](lines7-11).
Algorithm1:TrainingofST-ResNet
Input:Historicalobservations:{X ,···,X };
0 n−1
externalfeatures:{E ,···,E };
0 n−1
lengthsofcloseness,period,trendsequences:l ,l ,l ;
c p q
peroid: p;trendspan:q.
Output:ST-ResNetmodelM.
//constructtraininginstances
1 D←−∅
2 forallavailabletimeintervalt(1≤t≤n−1)do
43 SScp==[[XXtt−−llcp,·pX,Xt−(tl−c−(l1p)−,1·)··p·,,·X··t,−X1]t−p]
5 Sq=[Xt−lq·q,Xt−(lq−1)·q,···,Xt−q]
//Xtisthetargetattimet
6 putantraininginstance({Sc,Sp,Sq,Et},Xt)intoD
//trainthemodel
7 initializetheparametersθ
8 repeat
9 randomlyselectabatchofinstancesDbfromD
10 findθbyminimizingtheobjective(6)withDb
11 untilstoppingcriteriaismet
12 outputthelearnedST-ResNetmodelM
Aftertraining,thelearnedST-ResNetmodelMisobtainedforthesingle-ormulti-steplook-aheadprediction.the
processofwhichissummarizedinAlgorithm2. Sometypesofexternalfeatures(i.e.,weather)usedherearedifferent
fromthatinAlgorithm1. Inthetrainingprocess, weusethetrueweatherdata, whichisreplacedbytheforecasted
weatherdatainAlgorithm2.
Algorithm2:Multi-stepAheadPredictionUsingST-ResNet
Input: LearnedST-ResNetmodel:M;
numberoflook-aheadsteps:k;
historicalobservations:{X ,···,X };
0 n−1
externalfeatures:{En,···,En+k−1};
lengthsofcloseness,period,trendsequences:l ,l ,l ;
c p q
peroid: p;trendspan:q.
1 X←−{X0,···,Xn−1}//(i.e.,Xt=Xt,∀t)
2 fort=nton+k−1do
43 SScp==[[XXtt−−llcp,·pX,Xt−(tl−c−(l1p)−,1·)··p·,,·X··t,−X1]t−p]
5 Sq=[Xt−lq·q,Xt−(lq−1)·q,···,Xt−q]
6 X(cid:98)t ←−M(Sc,Sp,Sq,Et)
7 putX(cid:98)tintoX,i.e.,Xt =X(cid:98)t
8 output{X(cid:98)n,···,X(cid:98)n+k−1}
10