Statistical Tools for Program Evaluation : Methods and Applications to Economic Policy, Public Health, and Education

530 Pages·2017·39.33 MB·English

by  Jean-Michel Josselin

Jean-Michel Josselin
Benoît Le Maux
Statistical Tools
for Program
Evaluation
Methods and Applications to Economic
Policy, Public Health, and Education
Statistical Tools for Program Evaluation
Jean-Michel Josselin (cid:129) Benoˆıt Le Maux
Statistical Tools for
Program Evaluation
Methods and Applications to Economic
Policy, Public Health, and Education
Jean-MichelJosselin BenoˆıtLeMaux
FacultyofEconomics FacultyofEconomics
UniversityofRennes1 UniversityofRennes1
Rennes,France Rennes,France
ISBN978-3-319-52826-7 ISBN978-3-319-52827-4 (eBook)
DOI10.1007/978-3-319-52827-4
LibraryofCongressControlNumber:2017940041
#SpringerInternationalPublishingAG2017
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
or information storage and retrieval, electronic adaptation, computer software, or by similar or
dissimilarmethodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexempt
fromtherelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinformationinthis
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authors or the editors give a warranty, express or implied, with respect to the material contained
hereinor for anyerrors oromissionsthat may havebeenmade. Thepublisher remainsneutralwith
regardtojurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
Printedonacid-freepaper
ThisSpringerimprintispublishedbySpringerNature
TheregisteredcompanyisSpringerInternationalPublishingAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Acknowledgments
We would like to express our gratitude to those who helped us and made the
completionofthisbookpossible.
Firstofall,wearedeeplyindebtedtotheSpringereditorialteamandparticularly
Martina BIHN whose support and encouragement allowed us to finalize this
project.
Furthermore, we have benefited from helpful comments by colleagues and we
wouldliketoacknowledgethehelpofMauriceBASLE´,ArthurCHARPENTIER,
Pauline CHAUVIN, Salah GHABRI, and Christophe TAVE´RA. Of course, any
mistakethatmayremainisourentireresponsibility.
In addition, we are grateful to our students who have been testing and
experimenting our lectures for so many years. Parts of the material provided here
havebeentaughtattheBachelorandMasterlevels,inFranceandabroad.Several
students and former students have been helping us improve the book. We really
appreciated their efforts and are very grateful to them: Erwan AUTIN, Benoˆıt
CARRE´,AudeDAILLE`RE,Kristy´naDOSTA´LOVA´,andAdrienVEZIE.
Finally, we would like to express our sincere gratefulness to our families for
theircontinuoussupportandencouragement.
v
Contents
1 StatisticalToolsforProgramEvaluation:Introductionand
Overview.. . . . . . .. . . . . . .. . . . . . .. . . . . . . .. . . . . . .. . . . . . .. 1
1.1 TheChallengeofProgramEvaluation. . . . . . . . . . . . . . . . . . . 1
1.2 IdentifyingtheContextoftheProgram. . . . . . . . . . . . . . . . . . 4
1.3 ExanteEvaluationMethods. . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.4 ExpostEvaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.5 HowtoUsetheBook?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Bibliography. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
PartI IdentifyingtheContextoftheProgram
2 SamplingandConstructionofVariables. . . .. . . . . . . .. . . . . . . .. 15
2.1 AStepNottoBeTakenLightly. . . . . . . . . . . . . . . . . . . . . . . 15
2.2 ChoiceofSample. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3 ConceptionoftheQuestionnaire. . . . . . . . . . . . . . . . . . . . . . . 22
2.4 DataCollection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.5 CodingofVariables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Bibliography. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3 DescriptiveStatisticsandIntervalEstimation. . . . . . . . . . . . . . . . . 45
3.1 TypesofVariablesandMethods. . . . . . . . . . . . . . . . . . . . . . . 45
3.2 TabularDisplays. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.3 GraphicalRepresentations. . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.4 MeasuresofCentralTendencyandVariability. . . . . . . . . . . . . 64
3.5 DescribingtheShapeofDistributions. . . . . . . . . . . . . . . . . . . 69
3.6 ComputingConfidenceIntervals. . . . . . . . . . . . . . . . . . . . . . . 77
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4 MeasuringandVisualizingAssociations. . . . . . . . . . . . . . . . . . . . . 89
4.1 IdentifyingRelationshipsBetweenVariables. . . . . . . . . . . . . . 89
4.2 TestingforCorrelation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
4.3 Chi-SquareTestofIndependence. . . . . . . . . . . . . . . . . . . . . . 99
vii
viii Contents
4.4 TestsofDifferenceBetweenMeans. . . . . . . . . . . . . . . . . . . . . 105
4.5 PrincipalComponentAnalysis. . . . . . . . . . . . . .. . . . . . . . . . . 113
4.6 MultipleCorrespondenceAnalysis. . . . . .. . . . . . .. . . . . . . .. 126
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
5 EconometricAnalysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
5.1 UnderstandingtheBasicRegressionModel. . . . . . . . . . . . . . . 137
5.2 MultipleRegressionAnalysis. . . . . . . . . . . . . . . . . . . . . . . . . 147
5.3 AssumptionsUnderlyingtheMethodofOLS. . . . . . . . . . . . . . 153
5.4 ChoiceofRelevantVariables. . . . . . . . . . . . . . . . . . . . . . . . . 156
5.5 FunctionalFormsofRegressionModels. . . . . . . . . . . . . . . . . 164
5.6 DetectionandCorrectionofEstimationBiases. . . . . . . . . . . . . 167
5.7 ModelSelectionandAnalysisofRegressionResults. . . . . . . . 174
5.8 ModelsforBinaryOutcomes. . . . . . . . . . . . . . . . . . . . . . . . . . 180
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
6 EstimationofWelfareChanges. . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
6.1 ValuingtheConsequencesofaProject. . . . . . . . . . . . . . . . . . 189
6.2 ContingentValuation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
6.3 DiscreteChoiceExperiment. . . . . . . . . . . . . . . . . . . . . . . . . . 200
6.4 HedonicPricing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
6.5 TravelCostMethod. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
6.6 Health-RelatedQualityofLife. . . . . . . . . . . . . . . . . . . . . . . . 221
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
PartII ExanteEvaluation
7 FinancialAppraisal. . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . .. . 235
7.1 MethodologyofFinancialAppraisal. . . . . . . . . . . . . . . . . . . . 235
7.2 TimeValueofMoney. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
7.3 CashFlowsandSustainability. . . . . . . . . . . . . . . . . . . . . . . . . 244
7.4 ProfitabilityAnalysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
7.5 RealVersusNominalValues. . . . . . . . . . . . . . . . . . . . . . . . . . 255
7.6 RankingInvestmentStrategies.. . . . . . . . . . .. . . . . . . . . . .. . 257
7.7 SensitivityAnalysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
8 BudgetImpactAnalysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
8.1 IntroducingaNewInterventionAmongstExistingOnes. . . . . . 269
8.2 AnalyticalFramework. . . . .. . . . . .. . . . . .. . . . . . .. . . . . .. 271
8.3 BudgetImpactinaMultiple-SupplySetting. . . . . . . . . . . . . . . 275
8.4 Example. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. 277
8.5 SensitivityAnalysiswithVisualBasic. . . . . . . . . . . . . . . . . . . 281
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
Contents ix
9 CostBenefitAnalysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
9.1 RationaleforCostBenefitAnalysis. . . . . . . . . . . . . . . . . . . . . 291
9.2 ConceptualFoundations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
9.3 DiscountofBenefitsandCosts. . . . . . . . . . . . . . . . . . . . . . . . 299
9.4 AccountingforMarketDistortions. . .. . . . . . . . .. . . . . . . . .. 306
9.5 DeterministicSensitivityAnalysis. . . . . . . . . . . . . . . . . . . . . . 311
9.6 ProbabilisticSensitivityAnalysis. . . . . . . . . . . . . . . . . . . . . . . 313
9.7 Mean-VarianceAnalysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
Bibliography. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324
10 CostEffectivenessAnalysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
10.1 AppraisalofProjectswithNon-monetaryOutcomes. . . . . . . . . 325
10.2 CostEffectivenessIndicators. . . . . . . . . . . . . . . . . .. . . . . . . . 328
10.3 TheEfficiencyFrontierApproach. . . . . . . . . . . . . . . . . . . . . . 336
10.4 DecisionAnalyticModeling. . . . . . . . . . . . . . . . . . . . . . . . . . 342
10.5 NumericalImplementationinR-CRAN. . . . . . . . . . . . . . . . . . 351
10.6 ExtensiontoQALYs. . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . 357
10.7 UncertaintyandProbabilisticSensitivityAnalysis. . . . . . . . . . 358
10.8 AnalyzingSimulationOutputs. . . . . . . . . . . . . . . . . . . . . . . . . 371
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382
11 Multi-criteriaDecisionAnalysis. . . . . . . . . . . . . . . . . . . . . . . . . . . 385
11.1 KeyConceptsandSteps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385
11.2 ProblemStructuring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 388
11.3 AssessingPerformanceLevelswithScoring. . . . . . . . . . . . . . . 390
11.4 CriteriaWeighting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395
11.5 ConstructionofaCompositeIndicator. . . . . . . . . . . . . . . . . . . 398
11.6 Non-CompensatoryAnalysis. . . . . . . . . . . . . . . . . . . . . . . . . . 401
11.7 ExaminationofResults. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416
PartIII ExpostEvaluation
12 ProjectFollow-UpbyBenchmarking. . . . . . . . . . . . . . . . . . . . . . . 419
12.1 CostComparisonstoaReference. . . . . . . . . . . . . . . . . . . . . . 419
12.2 CostAccountingFramework. . . . . . . . . . . . . . . . . . . . . . . . . . 423
12.3 EffectsofDemandStructureandProductionStructure
onCost. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426
12.4 ProductionStructureEffect:Service-OrientedApproach. . . . . . 433
12.5 ProductionStructureEffect:Input-OrientedApproach. . . . . . . 436
12.6 RankingThroughBenchmarking. . . . . . . . . . . . . . . . . . . . . . . 440
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
x Contents
13 RandomizedControlledExperiments. . . . . . . . . . . . . . . . . . . . . . . 443
13.1 FromClinicalTrialstoFieldExperiments. . . . . . . . . . . . . . . . 443
13.2 RandomAllocationofSubjects. . . . . . . . . . . . . . . . . . . . . . . . 448
13.3 StatisticalSignificanceofaTreatmentEffect. . . . . . . . . . . . . . 453
13.4 ClinicalSignificanceandStatisticalPower. . . . . . . . . . . . . . . . 463
13.5 SampleSizeCalculations. . . . . . . . . . . . . . . . . . . . . . . . . . . . 471
13.6 IndicatorsofPolicyEffects. . . . . . . . . . . . . . . . . . . . . . . . . . . 474
13.7 SurvivalAnalysiswithCensoring:TheKaplan-Meier
Approach. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480
13.8 Mantel-HaenszelTestforConditionalIndependence. .. . . . . .. 483
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 487
14 Quasi-experiments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 489
14.1 TheRationaleforCounterfactualAnalysis. . . . . . . . . . . . . . . . 489
14.2 Difference-in-Differences. . . . . . . . . . . . . . . . . . . . . . . . . . . . 492
14.3 PropensityScoreMatching. . . . . . . . . . . . . . . . . . . . . . . . . . . 498
14.4 RegressionDiscontinuityDesign. . . . . . . . . . . . . . . . . . . . . . . 512
14.5 InstrumentalVariableEstimation. . . . . . . . . . . . . . . . . . . . . . . 519
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 530
1
Statistical Tools for Program Evaluation:
Introduction and Overview
1.1 The Challenge of Program Evaluation
Thepast30yearshaveseenaconvergenceofmanagementmethodsandpractices
betweenthepublicsectorandtheprivatesector,notonlyatthecentralgovernment
level (in particular in Western countries) but also at upper levels (European
commission, OECD, IMF, World Bank) and local levels (municipalities, cantons,
regions). This “new public management” intends to rationalize public spending,
boosttheperformanceofservices,getclosertocitizens’expectations,andcontain
deficits.Akeyfeatureofthisevolutionisthatprogramevaluationisnowadayspart
ofthepolicy-makingprocessor,atleast,onitswayofbecominganimportantstep
in the design of public policies. Public programs must show evidence of their
relevance, financial sustainability and operationality. Although not yet systemati-
callyenacted,programevaluationintendstograsptheimpactofpublicprojectson
citizens,ascomprehensivelyaspossible,fromeconomictosocialandenvironmen-
talconsequencesonindividualandcollectivewelfare.Ascanbededuced,thetask
is highly challenging as it is not so easy to put a value on items such as welfare,
health, education or changes in environment. The task is all the more demanding
that a significant level of expertise is required for measuring those impacts or for
comparingdifferentpolicyoptions.
Thepresentchapteroffersanintroductiontothemainconceptsthatwillbeused
throughout the book. First, we shall start with defining the concept of program
evaluationitself.Althoughthereisnoconsensusinthisrespect,wemayrefertothe
OECDglossarywhichstatesthatevaluationisthe“processwherebytheactivities
undertaken by ministries and agencies are assessed against a set of objectives or
criteria.” According to Michael Quinn Patton, former President of the American
EvaluationAssociation,programevaluationcanalsobedefinedas“thesystematic
collection of information about the activities, characteristics, and outcomes of
programs, for use by people to reduce uncertainties, improve effectiveness, and
makedecisions.”Wemayalsoproposeourowndefinitionoftheconcept:program
evaluationisaprocessthatconsistsincollecting,analyzing,andusinginformation
#SpringerInternationalPublishingAG2017 1
J.-M.Josselin,B.LeMaux,StatisticalToolsforProgramEvaluation,
DOI10.1007/978-3-319-52827-4_1