Mapping Policy Preferences from Texts: Statistical Solutions for Manifesto Analysts PDF
Preview Mapping Policy Preferences from Texts: Statistical Solutions for Manifesto Analysts
OUPCORRECTEDPROOF–FINAL,24/10/2013,SPi MAPPING POLICY PREFERENCES FROM TEXTS OUPCORRECTEDPROOF–FINAL,24/10/2013,SPi OUPCORRECTEDPROOF–FINAL,24/10/2013,SPi Mapping Policy Preferences From Texts Statistical Solutions For Manifesto Analysts ANDREA VOLKENS JUDITH BARA IAN BUDGE MICHAEL D. MCDONALD HANS-DIETER KLINGEMANN WITH ROBIN E. BEST SIMON FRANZMANN ONAWA P. LACEWELL POLA LEHMANN NICOLAS MERZ THOMAS MEYER SVEN REGEL HENRIKE SCHULTZE ANNIKA WERNER 1 OUPCORRECTEDPROOF–FINAL,24/10/2013,SPi 3 GreatClarendonStreet,Oxford,OX26DP, UnitedKingdom OxfordUniversityPressisadepartmentoftheUniversityofOxford. ItfurtherstheUniversity’sobjectiveofexcellenceinresearch,scholarship, andeducationbypublishingworldwide.Oxfordisaregisteredtrademarkof OxfordUniversityPressintheUKandincertainothercountries ©theseveralcontributors2013 Themoralrightsoftheauthorshavebeenasserted FirstEditionpublishedin2013 Impression:1 Allrightsreserved.Nopartofthispublicationmaybereproduced,storedin aretrievalsystem,ortransmitted,inanyformorbyanymeans,withoutthe priorpermissioninwritingofOxfordUniversityPress,orasexpresslypermitted bylaw,bylicenceorundertermsagreedwiththeappropriatereprographics rightsorganization.Enquiriesconcerningreproductionoutsidethescopeofthe aboveshouldbesenttotheRightsDepartment,OxfordUniversityPress,atthe addressabove Youmustnotcirculatethisworkinanyotherform andyoumustimposethissameconditiononanyacquirer PublishedintheUnitedStatesofAmericabyOxfordUniversityPress 198MadisonAvenue,NewYork,NY10016,UnitedStatesofAmerica BritishLibraryCataloguinginPublicationData Dataavailable ISBN 978–0–19–964004–1 PrintedinGreatBritainby CPIGroup(UK)Ltd,Croydon,CR04YY LinkstothirdpartywebsitesareprovidedbyOxfordingoodfaithand forinformationonly.Oxforddisclaimsanyresponsibilityforthematerials containedinanythirdpartywebsitereferencedinthiswork. OUPCORRECTEDPROOF–FINAL,24/10/2013,SPi Foreword MEASUREMENT RECOMMENDATIONS Theserecommendationscomerightupfrontbecausetheyareofprimeinterestto Manifesto analysts seeking solutions for their research problems. They leapfrog supporting discussion to give direct practical advice based on our statistical conclusions. From these, analysts can see at a glance what the ‘state of the data’ is,howtheyareadvisedtoproceedandwhatsupportingmeasurestheycandraw on.Ofcourseanyadviceneedsthebackupprovidedbyrelevantchapters.Butthe summaryshouldhelpreaderscutthroughtothem General advice must of course be tailored to particular research objectives. However,thecarewehavetakentoconfrontalltheproblemscommonlyraisedby usersandcriticsofthedatasetshouldrenderourcounselsrelevanttomostanalyses overavarietyofsubfields.Evidenceforsuchconclusionsandadvicewillfollow.Here wesimplyprovidethesubstanceintheshapeofthesenumberedrecommendations: 1. Takenasawhole,i.e.assummarizedintheleft-rightscaleorwhereall56,or substantial subsets, of the policy categories to which quasi-sentences are assigned are input together, the dataset has high validity and reliability (80–100%). Estimates at this level are best input to statistical routines (e.g. regression or dimensional analyses) as they stand, without distorting adjustments—for example through the ‘errors in variable’ procedure in Stataviatheeivregcommand(Chapter4). 2. Sub-scales (free market, welfare, economy, peace, European Union), and most of the original policy categories are also best input individually into statisticalroutineswithoutprioradjustment. 3. Reliability and error coefficients should not be fed into multivariate analyses for the Manifesto estimates alone as this distorts results (Table4.1). Either all variables should be adjusted for reliability or none. 4. Proper caution should be exercised where some of the original categories are verythinlypopulatedorambiguousintermsofyourresearchpurposes.They shouldnotthenbeusedasvariablesoraspointestimatesontheirown,butonlyas partoftheoveralldatasetorasindirectcontributorstotheleft-rightscale. 5. Apart from such cases, the original estimates should also be used when comparingdistributionsacrosstimeorspace,withoutadjustment. 6. The original left-right point estimates will not generally give misleading results in comparisons of adjacent party or other policy positions over timeorspace,providingthereisreasonablediscountingofsmalldifferences. 7. Aguidetosuchdiscountingisprovidedbytheconfidenceintervalsreported forvariouslevelsofanalysisinChapter6. 8. Exact confidence intervals for each of 80 parties over all 60 variables are reportedontheMARPORwebsite(https://manifesto-project.wzb.eu)for eachelection,ifthisdegreeofprecisionisrequired. OUPCORRECTEDPROOF–FINAL,24/10/2013,SPi vi Foreword 9. ReliabilitymeasuresandconfidenceintervalsreportedontheMARPORwebsite arebasedonobservedstabilityandchangeintheManifestopolicyestimates includedinthedataset(MPD ).Theythuscaptureallthetypesoferrorwhich b affectestimation,fromdocumentselectiontocoding,transcription,etc. 10. Final estimate measures of uncertainty and error should always be used in preference to measures which make stronger assumptions about the natureofthedocumentsortheirselectionorpreparation,sincetheseare oftenwrongand/orleadtoparadoxicalconsequences(Chapters4and5). Inparticular,thelengthofadocumentisnoguidetoitsreliability,given a) ambiguity about what length implies in terms of ‘noise’, and b) the different types of document used to base estimates on, where the significance of length varies. 11. Final estimate-based measures are also the most relevant for researchers, whosemainconcerniswiththepartypolicyprofileineachelection.Thusit isthereliabilityandvalidityofpolicyindicatorsassuch—notofproceduresfor producing them—which are of most interest. These are measured directly throughthefinal-estimateapproach. 12. Comparisons across extended stretches of space and time can only be made with invariant indicators which are deductively rather than inductively derived. Changing the left-right scale or the original coding scheme would destroythe possibilityof such extendedcomparisons. 13. Being derived from the ideological divisions around 1900 which produced modern party systems, the standard left-right scale (RILE) has a continuing contemporary relevance which is likely to continue into the future. 14. None of these caveats should prevent researchers devising alternative measure for their own purposes, as the data are capable of supporting almost any number of combinations and re-combinations adapted to various research uses. Only, they should bear in mind that these are likely to be severely limited over time and space and so will not serve as replacements for the MRG-CMP standard measures. This is particularly trueifalternativemeasuresareinductivelyderivedfromsubsetsofthedata. 15. In comparing party positions estimated from the Manifesto data to electoral or other policy positions estimated from mass or expert surveys, analysts should ‘re-centre’ the latter as they miss the cross-national variation which the Manifesto estimates pick up (Chapter2). Without ‘re-centring’,likeisnotbeingcomparedwithlike. 16. MPD ,thedatabasecreatedbyMARPOR,facilitatesmultilevel,comparative, b andover-timeanalysisbyincorporatingsuchtransformationsandpre-linking data in commonly requested combinations (see Table1.1). All measures, estimates, and supporting macrodata are available on open access, with currently processed data instantaneously available. A particularly useful feature is its ability to retrieve and supply all datasets previously requested, sothattheactualdataanalysedforresearchreportsandotherusesisavailable forre-analysisandchecks(Chapter10). OUPCORRECTEDPROOF–FINAL,24/10/2013,SPi Preface INFORMATION OR ERROR? ANSWERING THE CORE QUESTION PLURALISTICALLY Staring into a gift horse’s mouth while ignoring its winning performance is a proverbialmistake.TheManifestodatasethasbeenarealgiftforpoliticalscience, permitting empirical analysis of key policy relationships which had previously been matters for speculation. Its winning performance is attested by an APSA (American Political Science Association) award and prime place in around 300 research publications. The current extension of the estimates to Latin America promisesfurtherbenefits. Usershave,however,beenconfusedbycritiqueswhichfocusonerroranduncer- tainty—foundtosomeextentinanysetofstatisticalmeasures—ratherthanontheir attestedvalidityacrossmanyfieldsofpoliticalscience.Inconsideringerrorthecore question is whether the variation in policy positions revealed by the Manifesto estimates—and not found in party familycharacterizations or electoral and expert judgements—is real information about policy differences or simply a reflection of differenttextcodingsproducedbydifferentpeopleatdifferenttimes.Truedifferences andmeasurementerrorsareundoubtedlypresentinalldata.Sothekeymethodo- logical question can be phrased more precisely as ‘How can we determine when variationinManifestoestimatesreflectserrorasopposedtorealdifferencesinpolicy?’ With data collected under three regimes—the Manifesto Research Group (MRG) (1979–1983), Comparative Manifesto Project (CMP) (1983–2009), and Manifesto Research on Political Representation (MARPOR) (2009 and continu- ing)—andconstantlyexpandingoverspaceandtime,itisclearthatnoonetestor setoffigurescanprovideadefinitiveanswerforthewholeofthepresentdataset— orconceivablefutureones.Thecaseforwide,contextualevaluationisreinforced bythefactthatdifferentresearchobjectivesgeneratedifferentconceptionsofwhat isreliableandvalidinthefirstplace(Klingemannetal.2006,Chapters4and5). Thecontinuityofmeasurementprocedures(documentselection,codingframe, collective coding, scale construction) over different places and times does allow some methodological generalization from one phase of the Manifesto dataset to another.Howeverweshouldbechary,forexampleaboutsimplyextrapolatingthe excellentinter-coderagreementobtainedin1981–1983fortheMRGphaseofthe data (Budge, Robertson, Hearl, eds, 1987) to the updated and vastly expanded dataset of today. The Markovian assumptions of the Heise (1969) reliability test (Klingemannetal.2006,91–2)applytosomeperiodscoveredbythedatasetbut nottoothers.Otherreliabilitychecks(e.g.Hausman1978)canbeappliedoverthe wholeset(McDonald2006:89)—butonlyonparticular,ifplausible,assumptions. Difficulties with other reliability measures and derived confidence intervals are compoundedwhentheirbasicassumptionsaresuspect(Chapters4–5). In these circumstances the repeated validation of the estimates reported for different times and places, different phases of the dataset, and under different methods of analysis, has to have prime weight in any truth assessment. The OUPCORRECTEDPROOF–FINAL,24/10/2013,SPi viii Preface continuity of the estimation procedures which produced the Manifesto data allows unique checks to be made against historical experience and theoretical expectations across vast swathes of space and time. The extent to which the originalestimatesmatchuptothesemustformthebasicevidencefortheirhigh informationcontent. Unfortunately,theseconsiderationsseemlostonsomecommentators,whowill accept no evidence on the informational value of the data not provided by their own, often flawed, error measures. On that basis they have proposed major adjustmentstomeasuresandestimates,arguingthatifcorrecttheyimprovedata qualityandifwrongdonotdamageit.Unfortunately,thisargumentisfallacious. AsChapters4and5demonstrate,datainterventionscarryheavycosts,especially when based on controvertible assumptions. Far better build confidence intervals and other uncertainty measures on a minimalist basis and let researchers tailor themtothetaskinhand,whichmayrequiremore,orless,restrictiveassumptions aboutuncertainty(Chapter6). ThisbookempowersusersoftheManifestoestimates—andtextualanalystsin general—byprovidingthemwiththeessentialinformationtochooseameasure- ment approachthatworksbestforthem: sticking totheoriginal estimatesgiven theirextensivevalidation,usingreliabilitycoefficientsanderrortestsgroundedon documentedcharacteristicsofthedata,andavoiding‘legislation’promulgatedby mistakencommentators. Thefollowingchaptersspellthisout.Fornowwewishtoacknowledge,firstand foremost,theGermanResearchFoundation(DFG)forhavingtheperspicacityto institute a programme of long-term research grants from which MARPOR has benefitted greatly, and the Nuffield Foundation for their support of work which hascontributedtotheproject. We are immensely grateful for the constructive role of our editor, Dominic Byatt,andeditorialassistants,LizzieSufflingandAimeeWrightandtheirteamat OxfordUniversityPressaswellastotheanonymousreadersofthemanuscriptfor their helpful comments. Special thanks go to Linda Day for her patience and steadfastnessinhelpingpreparethemanuscriptforpublication. We appreciate the encouragement we have received from Professor Dr Wolf- gang Merkel, Director of Research Unit ‘Democracy: Structures, Performance, Challenges’ at the Wissenschaftszentrum Berlin fur Sozialforschung. All the student assistants who have worked on MARPOR since 2009, Donald Blondin, Agata Chroboczek, Daniel Drewski, Timm Frerk, Paula Glamann, Jonathan Homola,CalineIttner,VerenaKröss,MariaNößler,BenjaminRestle,andDaniela Russhaveprovidedinvaluablecontributionstoitsdevelopmentaswellashelping uswithdatapreparationforthebookwhichwealsogratefullyacknowledge.We would like to thank Ben Farrer, M. Steen Thomas, and Josh Zingher and the CentreonDemocraticPerformanceatBinghamtonUniversity,StateUniversityof New York for their assistance in this enterprise as well as Lawrence Ezrow and HershbinderMannoftheUniversityofEssex. Finally,wenotethecontributionsmadebythemanyusersofthedatasinceitfirst becameavailablein2001.Thesehaveledtomanypublicationswhichhavebrought exciting new ideas and developments in analysis that have not only contributed much to academic debate but have also enriched the scope and reliability of the material,andhavecontributedgreatlytothisbook.Wetrustthatthiswillcontinue farintothefutureasMARPORextendsitsrangeofmaterial. OUPCORRECTEDPROOF–FINAL,24/10/2013,SPi Contents ListofFigures xi ListofTables xiii ListofAbbreviations xvii Introduction:CharacterizingtheDataCorrectlyinorderto MeasurethemAccurately 1 PARTI VALIDATED,AUTHORITATIVE,INDISPENSABLE:THE MANIFESTOESTIMATESINPOLITICALRESEARCH 1. TheBestToolstoTackletheJob 9 IanBudgeandThomasMeyer 2. UsingtheManifestoEstimatestoCorrectSystematic‘Centring’Error inExpertandElectoralPositioningofParties 33 RobinE.Best 3. UsingtheManifestoEstimatestoRefinePartyFamilyPlacements 49 Hans-DieterKlingemannandIanBudge PARTII VALIDITYGUARANTEESRELIABILITY: HIGHRELIABILITYLIMITSERROR 4. ValidatedEstimatesversusDodgyAdjustments:FocusingExcessively onErrorDistortsResults 69 IanBudge,MichaelD.McDonald,andThomasMeyer 5. UnderstandingandValidatingtheLeft-RightScale(RILE) 85 IanBudgeandThomasMeyer 6. MeasuringUncertaintyandErrorDirectlyfromEndEstimates 107 MichaelD.McDonald PARTIII DELIVERINGQUALITYDATA: COLLECTION—CODING—CONTROLS—COMMUNICATION 7. LinkingUncertaintyMeasurestoDocumentSelectionandCoding 131 IanBudge 8. WhatareManifestosfor?SelectingandTypingDocumentsin theDatabase 146 NicolasMerzandSvenRegel 9. CoderTraining:KeytoEnhancingReliabilityandValidity 169 OnawaP.LacewellandAnnikaWerner
Description:The list of books you might like
