Table Of Content

Forecast Verification Forecast Verification A Practitioner’s Guide in Atmospheric Science Editedby IAN T. JOLLIFFE UniversityofAberdeen and DAVID B. STEPHENSON UniversityofReading Copyright(cid:1)2003John Wiley&SonsLtd,TheAtrium,Southern Gate,Chichester, West SussexPO198SQ,England Telephone((cid:1)44)1243779777 e-mail(for ordersandcustomerserviceenquiries):[email protected] Visit our HomePageonwww.wileyeurope.comorwww.wiley.com AllRightsReserved.Nopart ofthispublicationmaybereproduced,storedin a retrieval systemortransmitted in anyformorbyanymeans,electronic,mechanical,photocopying, recording,scanningor otherwise,except underthetermsoftheCopyright,Designsand PatentsAct 1988orunderthetermsofalicenceissued bytheCopyright LicensingAgency Ltd,90TottenhamCourtRoad,LondonW1T4LP,UK,withoutthepermissioninwritingof thePublisher.RequeststothePublishershouldbeaddressedtothePermissionsDepartment, John Wiley&SonsLtd,TheAtrium,Southern Gate,Chichester,West SussexPO198SQ, England,[email protected],orfaxedto((cid:1)44)1243770620. Thispublication isdesignedto provideaccurateandauthoritativeinformation inregard to thesubjectmattercovered.ItissoldontheunderstandingthatthePublisherisnotengagedin renderingprofessionalservices.Ifprofessionaladviceorother expert assistanceisrequired, theservicesofacompetent professionalshould besought. O t her W iley E dit orial O ffices John Wiley&SonsInc.,111River Street,Hoboken,NJ07030,USA Jossey-Bass,989Market Street,San Francisco,CA94103-1741,USA Wiley-VCH VerlagGmbH,Boschstr.12,D-69469Weinheim,Germany John Wiley&SonsAustralia Ltd,33ParkRoad,Milton,Queensland 4064,Australia John Wiley&Sons(Asia)PteLtd,2ClementiLoop#02-01,JinXingDistripark, Singapore129809 JohnWiley&SonsCanadaLtd,22WorcesterRoad,Etobicoke,Ontario,CanadaM9W1L1 Wileyalsopublishesitsbooksinavarietyofelectronicformats.Somecontentthatappearsin print maynot beavailablein electronicbooks. Library ofCongressCataloging-in-PublicationData Forecast verification:apractitioner’sguidein atmosphericscience/edited byIan T. JolliffeandDavidB.Stephenson. p.cm. Includesbibliographicalreferencesand index. ISBN 0-471-49759-2(alk.paper) 1.Weatherforecasting–Statisticalmethods–Evaluation.I.Jolliffe,I.T.II.Stephenson, David B. QC996.5.F6772003 551.63–dc21 2002192424 BritishLibrary CataloguinginPublicationData Acataloguerecord for thisbookisavailablefromtheBritish Library ISBN 0-471-49759-2 Typesetin 10.5/13pt TimesNewRoman byKolamInformation ServicesPvt.Ltd, Pondicherry,India Printed andboundin Great Britain byAntonyRoweLtd,Chippenham,Wiltshire Thisbookisprintedonacid-freepaperresponsiblymanufacturedfromsustainableforestryin which at leasttwo treesareplantedfor each oneused forpaper production. Contents List of contributors ix Preface xi Chapter1 Introduction 1 I an T . J olliffe and D avid B . S t ephenson 1.1 A BriefHistoryand Current Practice 1 1.1.1 History 1 1.1.2 Current Practice 3 1.2 R ea sons for F oreca st Vecrai ftiion and its Benetfsi 4 1.3 T ypes of F oreca st s a nd Vcear itfiion D a ta 6 1.4 Scores, Skill, and Value 7 1. 4. 1 Sk ill Sco r es 8 1. 4. 2 A r ct iiaf il Sk ill 9 1. 4. 3 St a t ist ica l Sciga nn ifcie 10 1.4.4 ValueAdded 11 1.5 Data Qualityand Other PracticalConsiderations 11 Chapter2 BasicConcepts 13 JacquelineM. Potts 2.1 Introduction 13 2.2 TypesofPredictand 13 2.3 ExploratoryMethods 14 2.4 NumericalDescriptiveMeasures 19 2.5 Probability, Random Variablesand Expectations 23 2.6 Joint, Marginal, and ConditionalDistributions 24 2.7 Accuracy, Association and Skill 26 2.8 PropertiesofScoringRules 27 2. 9 Ver icfai tion as a R egression Problem 27 2.10 The M urphy–Winkler F ramework 29 2.11 D imensionalit y of t he Vecriaf ition Problem 36 Chapter3 BinaryEvents 37 IanB. Mason 3.1 Introduction 37 3. 2 Ver icfai tion M easures 39 3.2.1 SomeBasicDescriptiveStatistics 41 3.2.2 PerformanceMeasures 45 3. 3 Ver icfai tion of Binary F oreca st s: Theoretica l Considerations 56 vi Contents 3.3.1 A G eneral F ramework for Vifeicra tion: TheDistributions-oriented Approach 56 3.3.2 PerformanceMeasuresin TermsofFactorisations oftheJoint Distribution 58 3. 3. 3 M et a vecra itf iio n : C r it er ia f o r S cr een in g P er f o r m a n ce Measures 60 3.3.4 OptimalThresholdProbabilities 63 3 . 3 . 5 S a m p lin g U n cer t a in t yd ean nc ed InCt eor vna flis for PerformanceMeasures 64 3.4 SignalDetection Theoryand TheROC 66 3.4.1 TheSignalDetection Model 67 3.4.2 TheRelativeOperatingCharacteristic 68 3. 4. 3 Ver ciafi tion M easures on R OC Axes 71 3. 4. 4 Ver ciafi tion M easures F rom Signal D etect ion T heory 73 Chapter4 CategoricalEvents 77 Robert E. Livezey 4.1 Introduction 77 4.2 T he Contingency T able: N otation, Dn ietfiions and MeasuresofAccuracy 79 4. 2. 1 N o t a t io n a nn dit ioDn se fi 79 4.2.2 MeasuresofAccuracy 81 4.3 SkillScores 82 4.3.1 DesirableAttributes 82 4.3.2 Gandinand MurphyEquitableScores 84 4.3.3 GerrityEquitableScores 88 4.3.4 LEPSCAT 91 4.3.5 SummaryRemarksonScores 92 4.4 SamplingVariabilityoftheContingency Tableand SkillScores 93 Chapter5 ContinuousVariables 97 MichelDe´que´ 5.1 Introduction 97 5.2 Forecast Examples 97 5.3 First-order Moments 99 5.3.1 Bias 99 5.3.2 MeanAbsoluteError 101 5.3.3 Bia s Correct ion a nd Arctiifai l Sk ill 101 5.3.4 MeanAbsoluteErrorand Skill 102 5.4 Second and Higher-order Moments 103 5.4.1 MeanSquaredError 103 5.4.2 MSE SkillScore 104 5.4.3 MSE ofScaledForecasts 105 5.4.4 Correlation 106 5.4.5 AnExample:Testingthe‘Limit ofPredictability’ 110 5.4.6 Rank Correlations 110 5.4.7 ComparisonofMomentsoftheMarginal Distributions 113 Contents vii 5.5 ScoresBased on CumulativeFrequency 115 5.5.1 Linear Error in ProbabilitySpace 115 5.5.2 Quantile–QuantilePlots(q–qPlots) 116 5.5.3 ConditionalQuantilePlots 116 5.6 ConcludingRemarks 119 Chapter6 VerificationofSpatialFields 121 WasylDrosdowsky andHuqiangZhang 6.1 Introduction:TypesofFieldsand Forecasts 121 6.2 TemporalAveraging 124 6.3 SpatialAveraging 126 6.3.1 MeasuresCommonlyUsed intheSpatialDomain 126 6.3.2 MapTypingandAnalogueSelection 131 6.3.3 Accountingfor SpatialCorrelation 132 6.4 Assessment ofModelForecastsin the Spatio-temporalDomain 132 6.4.1 PrincipalComponent Analysis(EOF Analysis) 132 6.4.2 CombiningPredictabilitywith Model F o r eca st Vecr ai tf iio n 133 6.4.3 SignalDetectionAnalysis 134 6. 5 Ver icfai tion of Spatia l R ainfall F oreca st s 135 Chapter7 ProbabilityandEnsembleForecasts 137 ZoltanToth, OlivierTalagrand,Guillem Candille and Yuejian Zhu 7.1 Introduction 137 7.2 Main AttributesofProbabilisticForecasts 138 7.3 ProbabilityForecastsofBinaryEvents 142 7.3.1 TheReliabilityCurve 143 7.3.2 TheBrierScore 145 7. 3. 3 Ver ciafi t io n B a sed o n D ecisio n ProbabilityThresholds 149 7.4 ProbabilityForecastsofMoreThan Two Categories 151 7.4.1 Vector Generalization oftheBrier Score 151 7.4.2 Information Content asaMeasureofResolution 152 7.5 ProbabilityForecastsofContinuousVariables 154 7.5.1 TheDiscreteRankedProbabilityScore 154 7.5.2 TheContinuousRankedProbabilityScore 155 7.6 SummaryStatisticsfor EnsembleForecasts 155 7.6.1 EnsembleMeanErrorand Spread 156 7.6.2 EqualLikelihood FrequencyPlot 157 7.6.3 AnalysisRank Histogram 159 7.6.4 MultivariateStatistics 159 7.6.5 TimeConsistencyHistogram 161 7.7 LimitationsofProbabilityand Ensemble F o r eca st Vecr ai fti io n 162 7.8 ConcludingRemarks 162 viii Contents Chapter8 EconomicValueandSkill 165 DavidS. Richardson 8.1 Introduction 165 8.2 TheCost/LossRatio Decision Model 166 8.2.1 Valueofa DeterministicBinaryForecast System 168 8.2.2 ProbabilityForecasts 172 8.2.3 ComparisonofDeterministicandProbabilistic BinaryForecasts 175 8.3 TheRelationship Between Valueand theROC 176 8.4 OverallValueand theBrier SkillScore 180 8.5 Skill, Value, and EnsembleSize 183 8.6 Summary 186 Chapter9 ForecastVerification:Past, PresentandFuture 189 D avid B . S t ephenson and I an T . J olliffe 9.1 Introduction 189 9.2 ReviewofKeyConcepts 189 9.3 Forecast Evaluation in Other Disciplines 192 9.3.1 Statistics 192 9.3.2 Financeand Economics 194 9.3.3 Environmentaland EarthSciences 196 9.3.4 Medicaland ClinicalStudies 197 9.4 FutureDirections 198 Glossary 203 References 215 AuthorIndex 227 Subject Index 231 List of Contributors G. Candille LaboratoiredeMe´te´orologieDynamique, EcoleNormale Supe´rieure, 24RueLhomond, F 75231Pariscedex05, France. [email protected] M. De´que´ Meteo-FranceCNRM/GMGEC/EAC, 42Avenue Coriolis, 31057Toulousecedex01, France. [email protected] W. Drosdowsky BureauofMeteorologyResearchCentre,BMRC,POBox 1289K, Melbourne3001, Australia. w.drosdowsky@bom. gov.au I . T . J o lli ffe D ep a r t m en t o f M a t h em a t ica l S cien ces, U n iv er s it y o f Aberdeen, King’sCollege, Aberdeen AB243UE, UK. [email protected] R.E. Livezey W/OS4, ClimateServicesDivision, Room 13228, SSMC2, 1325East West Highway, Silver Spring, MD 20910–3283, USA. [email protected] I.B. M a son Canberra M eteorological Ocfefi, P O Bo x 797, C a n b er r a , ACT 2601, Australia. [email protected] J.M. Potts Biomathematicsand StatisticsScotland, TheMacaulay Institute, Craigiebuckler, Aberdeen AB158QH, UK. [email protected] D. Richardson MeteorologicalOffice,LondonRoad,Bracknell,Reading, RG122SZ, UK. [email protected] D.B. Stephenson Department ofMeteorology, UniversityofReading, EarleyGatePO Box243,ReadingRG66BB, UK. [email protected] O. Talagrand LaboratoiredeMe´te´orologieDynamique, EcoleNormale Supe´rieure, 24RueLhomond, F 75231Pariscedex05, France. [email protected] Z. Toth NOAAatNationalCentersforEnvironmentalPrediction, 5200Auth Rd., Room 207, Camp Springs, MD 20746, USA. [email protected] H. Zhang Bureau ofMeteorologyResearch Centre, BMRC, PO Box1289K, Melbourne3001, Australia. h.zhang@bom. gov.au x List of Contributors Y. Zhu NOAAatNationalCentersforEnvironmentalPrediction, 5200AuthRd.,Room207,CampSprings,MD20746,USA. [email protected]