International Journal of Environmental Research and Public Health Article Barriers to and Facilitators of the Evaluation of Integrated Community-Wide Overweight Intervention Approaches: A Qualitative Case Study in Two Dutch Municipalities TessaM.vanKoperen1,*,AnjadeKruif2,LisavanAntwerpen3,Anna-MarieHendriks4, JacobC.Seidell5,AlbertineJ.Schuit6andCarryM.Renders1 1 DepartmentofHealthSciences,FacultyofEarthandLifeSciences,SectionPreventionandPublicHealthand EMGO+InstituteforHealthandCareResearch,VUUniversityAmsterdam,DeBoelelaan1085, Amsterdam1081HV,TheNetherlands;[email protected] 2 DepartmentofHealthSciences,FacultyofEarthandLifeSciences,SectionMethodsandApplied Biostatistics,QualitativeResearch,VUUniversityAmsterdam,DeBoelelaan1085,Amsterdam1081HV, TheNetherlands;[email protected] 3 YouthonHealthWeight,Causariestraat5,DenHaag2511VB,TheNetherlands;[email protected] 4 DepartmentofHealthPromotion,FacultyofHealth,MedicineandLifeSciences,Maastricht, MaastrichtUniversity,Universiteitssingel60,Maastricht6229ER,TheNetherlands; [email protected] 5 DepartmentofHealthSciences,FacultyofEarthandLifeSciencesandtheEMGO+InstituteforHealthand CareResearch,VUUniversityAmsterdam,DeBoelelaan1085,Amsterdam1081HV,TheNetherlands; [email protected] 6 NationalInstituteforPublicHealthandtheEnvironment,POBox1,Bilthoven3720BA,TheNetherlands; [email protected] * Correspondence:[email protected];Tel.:+31-6-52678634 AcademicEditors:SaraKirk,Jessie-LeeMclsaacandTarraPenney Received:5February2016;Accepted:28March2016;Published:31March2016 Abstract: Topreventoverweightandobesitytheimplementationofanintegratedcommunity-wide interventionapproach(ICIA)isoftenadvocated. Evaluationcanenhanceimplementationofsuch an approach and demonstrate the extent of effectiveness. To be able to support professionals in the evaluation of ICIAs we studied barriers to and facilitators of ICIA evaluation. In this study tenprofessionalsoftwoDutchmunicipalitiesinvolvedintheevaluationofanICIAparticipated. Weconductedsemi-structuredinterviews(n=12),observedprogrammemeetings(n=4)andcarried outdocumentanalysis. Datawereanalyzedusingathematiccontentapproach. Welearnedthat evaluationishamperedwhenitisperceivedasunfeasibleduetolimitedtimeandbudget,alackof evaluationknowledgeoranegativeevaluationattitude. Otherbarriersareapoorunderstandingof theevaluationprocessanditsaddedvaluetooptimizingtheprogramme. Sufficientcommunication betweeninvolvedprofessionalsonevaluationcanfacilitateevaluation,asdoessupportforevaluation ofICIAstogetherwithstakeholdersatastrategicandtacticallevel. Tostimulatetheevaluationof ICIAs,werecommendsupportingprofessionalsinsecuringevaluationresources,providingtailored training and tools to enhance evaluation competences and stimulating strategic communication onevaluation. Keywords: evaluation;barriers;facilitators;perceptions;casestudy;community;overweight Int.J.Environ.Res.PublicHealth2016,13,390;doi:10.3390/ijerph13040390 www.mdpi.com/journal/ijerph Int.J.Environ.Res.PublicHealth2016,13,390 2of19 1. Introduction Childhoodobesityisaglobalpublichealthproblem[1,2]. In2013,23.8%oftheboysand22.6%of thegirlsindevelopedcountrieswereaffectedbyoverweightorobesityandindevelopingcountries thesepercentageswere12.9%and13.4%respectively[3]. Althoughtheincreaseintheprevalenceof childhoodobesityappearstobelevellingoffinsomecountries[4,5],inmostcountriestheincrease continues[6]. Obesityisassociatedwithanincreasedriskofseriouscomplicationssuchastype2 diabetesandcardiovasculardiseasesbutalsowithpsychosocialproblemsandareducedqualityof life[7,8]. Moreover,childhoodobesityoftenpersistsintoadulthood[9,10]. Governmentsandcommunitiesworldwidetrytopreventobesity,startinginchildhoodoreven beforebirth[11]. Traditionally,suchpreventionapproachesfocusedonindividualhealthdeterminants suchasthemotivationofchildrentobephysicallyactiveonadailybasis,orthelackofknowledge about healthy eating [12]. Unfortunately, these individually targeted prevention efforts showed disappointingoutcomes[13,14]. Thisseemedrelatedtothefactthattheseindividualdeterminantsof overweightandobesityweretargeted,whilethesocialandphysicalenvironmentinwhichchildren growupremainedthesameandthereforedidnotstimulateahealthierlifestyle[15]. Expertsarenow advocatingtheimplementationofanintegratedcommunity-wideinterventionapproach(ICIA),in whichbothpersonalaswellasenvironmentaldeterminantsaretargeted. Suchanapproachshould includeamixofinterventionsworkinginconjunctionwitheachotherinthesettingswherechildren live, learn, and play (early care and education, home, school, community, health care), should be directedtowardsmultipletargetgroupsinthecommunity,andshouldsharealong-termhealthrelated goal. Subsequently,itshouldbeimplementedbysectorswithinandoutsidethehealthdomain[16–20]. AsuccessfulexampleofanICIAistheEPODEapproach(EPODEstandsfor“EnsemblePrévenons l’ObésitéDesEnfants”or“TogetherLet’sPreventChildhoodObesity”),whichwasimplementedin 15countriesincludingtheNetherlands,whereitwasnamed“YouthonHealthyWeight”(InDutch: “Jongeren op Gezond Gewicht” or the JOGG-approach) [21–24]. The strength of this originally Frenchapproachliesinfivecriticalelements: (1)broadpoliticalcommitment(i.e.,fromhealthand non-health sectors) and integrated (public health) policies; (2) commitment of and collaboration betweenpublicandprivatestakeholders;(3)useofsocialmarketingtechniquesininterventiondesign; (4)scientificevaluationand(5)integratedpathwaysofpreventionandcare. Between2009and2014, 75municipalitiesadoptedthisapproachintheNetherlands,andthisnumbercontinuestorise.Toassist localprofessionalsinvolvedintheJOGG-approachwithdesigning,implementing,evaluatingand theadvocacyoftheapproachtowardsstakeholdersalogicmodelwasdeveloped,baseduponthe EPODElogicmodel[25](seeFigure1). OtherinstrumentsofferedbytheNationalCoordinationOffice oftheJOGG-approach(JOGG-office)tosupportprofessionalsintheevaluationoftheirICIAwere (1)anevaluationtrainingforprogrammemanagersandepidemiologists;(2)anevaluationmanual includingmultipleinstrumentsandtoolstosupporttheevaluationprocessanddatacollectionand (3)thepossibilityofinvolvinganevaluationexpert. Unfortunately,earlyexperienceswiththeJOGG-approachshowthatthecriticalelement“scientific evaluation”isoftenneglectedwithintheJOGG-approachesinmunicipalities,despitethefactthat evaluationisrecognizedasbeingimportantincontinuouslyimprovingtheICIAandinachievinga sustainableandmoreeffectiveprogramme[26,27]. EvaluationinthecontextofanICIAisbroaderthan simplyasystematicassessmentoftheworthormeritofsomething.Itis“thesystematicacquisitionand assessmentofinformationtoprovideusefulfeedbackaboutsomeobject”[28]. Hence,itrelatestothe totalprocessofformativeandsummativeevaluation,whichalreadycommenceswhenanintervention isinitiatedandcanbeusedtoimprovetheinterventionduringitscourse. InICIAsanevaluation processtakesplacewithinapoliticalandorganizationalcontext,itrequiresgroupskillsandcapacities, managementability,politicalleverageandsensitivitytotheneedsofmultiplestakeholders. Evaluation isthereforedifferentfromresearchwhichisgroundedinexperimentalmethodsandwhosegoalitisto createnewscientificknowledge. Aclassicalwaytodescribethisdifferencebetweenevaluationand researchisgivenbyPatton: “researchseekstoprove,evaluationseekstoimprove... ”[29]. Int.J.Environ.Res.PublicHealth2016,13,390 3of19 Int. J. Environ. Res. Public Health 2016, 13, 390 3 of 19 INPUT OUTPUT OUTCOME Contextual factors Political Support Intersectoral Changes in the Collaboration Environment Home Schools Care n Neighbourhoods ordination al organisatio Public Private Partnership CapCaocmitym Buuniitldy ing CHPhehiaylsdlthi cBya lNe Ahuctartiivvtioiitoyn u ↑↑r ChInilHcdWrreeeeaanilgst hwheyt i%t h a ntral CoOffice Loc Social Marketing ChIanndgiveisd uina lthe OGG Ce PInrteevgerantteiodn Paantdh wCaayres ANcPuAtihtvcryiitttisiiveoiicsntay ol& n AKAwntoSatiwtrkueileldnlsdeegsses J Scientific Guidance and Evaluation Figure 1. JOGG model. Figure1.JOGGmodel. Previous studies suggest that a possible barrier to the evaluation of complex community-wide Paprepvroioacuhsess tius dthiee sdisfuficgugletys tint hsealtecatipnog sasni balpepbroaprrriiaetre tdoestihgen eavndal muaettihoondoolofgcyo mforp elevxalucoatmiomn [u30n–i3ty3]-.w ide Other studies indicate that the evaluation of these initiatives is hampered because evaluation is approachesisthedifficultyinselectinganappropriatedesignandmethodologyforevaluation[30–33]. perceived differently by the various stakeholders involved [34,35] and performing such an evaluation Other studies indicate that the evaluation of these initiatives is hampered because evaluation is needs to compete with other priorities for scarce available resources (e.g., time, budget) [25,36,37]. perceiveddifferentlybythevariousstakeholdersinvolved[34,35]andperformingsuchanevaluation A limited number of studies have explored barriers to and facilitators of evaluation. However, to the needstocompetewithotherprioritiesforscarceavailableresources(e.g., time, budget)[25,36,37]. best of the authors’ knowledge no studies have involved professionals engaged in an ongoing A limited number of studies have explored barriers to and facilitators of evaluation. However, to ICIA [38–40]. thebestoIfnt ohredearu ttoh iomrsp’rokvneo swuplepdogrte ton othsotsue dimiepslehmavenetiinngv IoClvIAeds apcrroosfse scasisoesn, ainls-deenpgtha guenddeirnstaanndoinngg oing ICIAa[b3o8u–4t 0b]a.rriers to and facilitators of evaluation of ICIAs is needed. This understanding can then Iinmoprrodveer ptolainmnipnrgo vaneds uexpepcourtitotno otfh eovsaeluimatpiolne mofe tnhteisneg cIoCmIpArsehaecnrosisvse caapsperso,aicnh-edse apnthd uhenndceer mstaayn ding aboutinbfoarrmri eprusbtloic apnodlicyfa acnildit aptroarcsticoef oenv oaplutiamtiiozantioonf IoCf ItAhes uissen oefe IdCeIAd.s. TThheisreufonrde,e wrset aenxpdlionrgedc athne then perceptions of professionals involved in an ICIA with regards to barriers to and facilitators of the improveplanningandexecutionofevaluationofthesecomprehensiveapproachesandhencemay evaluation of an ongoing ICIA. informpublicpolicyandpracticeonoptimizationoftheuseofICIAs. Therefore,weexploredthe perceptions of professionals involved in an ICIA with regards to barriers to and facilitators of the 2. Materials and Methods evaluationofanongoingICIA. 2.1. Design 2. MaterialsandMethods A qualitative case study design was used to collect data in two Dutch municipalities (our cases) 2.1. Dthesaitg inmplemented the JOGG-approach. Since our goal was to explore a wide range of barriers and facilitators we chose to combine semi-structured interviews with observations and a document Aanqaluyaslisit aatmivoengca as ehsettuerdoygedneesoiugsn swamaspules eodf mtoucnoiclliepcatlidtiaest.a Tinhitsw doesDigunt cshupmpuorntisc itphae lietxiepslo(roautirvec ases) thatinmaptulreem oef nthteisd stthuedyJ O[4G1]G. T-ahpe pprrooapcohsa.lS winacse soubumrigttoeadl two athset omeexdpiclaolr eethaicwali dcoemramnigtteeeo offb tahrer iVeUrs and facilitmateodriscawl ececnhtorsee, wtohciochm jbuidngeesde mthia-ts tar uwctauivreedr oinf tmerevdieicwals ewthitihcaol baspeprrvoavtaiol nwsaasn adpapldicoacbulem feonrt tahnisa lysis amonegxpalohreatteivroe gsteundeyo uons psaemrceppletioonfsm ofu pnriocfiepsasliiotnieasls. aTbhoiustd eevsailgunatsiounp opfo IrCtsIAths.e explorativenatureofthis study[41]. TheproposalwassubmittedtothemedicalethicalcommitteeoftheVUmedicalcentre, 2.2. Sampling whichjudgedthatawaiverofmedicalethicalapprovalwasapplicableforthisexplorativestudyon perceptionTso orefcpruroitf eosusri otwnoa lcsaasebso, uwtee evmalauilaetdio an stoufdIyC pIAarst.icipation request to six eligible municipalities that had implemented the JOGG-approach in February 2014. Inclusion criteria were: one large and 2.2. Soanmep lsimngall municipality; with preferably different initiators of the JOGG-approach, which began implementation at least two years ago; affiliation with different Regional Public Health Services Torecruitourtwocases,weemailedastudyparticipationrequesttosixeligiblemunicipalities (RPHS); and no participation in another external research programme. Based on these criteria, we thathsaeldecitmedp tlwemo menutneidcipthaelitJiOesG (hGer-eaapfpterro caacllhedin CFaseeb Aru aanrdy C20as1e4 B.)I.n clusioncriteriawere: onelargeand one small municipality; with preferably different initiators of the JOGG-approach, which began implementation at least two years ago; affiliation with different Regional Public Health Services (RPHS);andnoparticipationinanotherexternalresearchprogramme. Basedonthesecriteria,we selectedtwomunicipalities(hereaftercalledCaseAandCaseB). Int.J.Environ.Res.PublicHealth2016,13,390 4of19 Fromthetwocases,weusedemergentsamplingtoselectaheterogeneousgroupofrespondents basedonposition(e.g.,programmemanagers,RPHSepidemiologists,representatives,policyadvisors) anddegreeofinvolvementwithintheJOGG-approach(e.g.,involvedindailyoperations,operating atstrategicorattacticallevel). Emergentsamplingimpliesthatourstudypopulationemergedand unfoldedwhilethestudyprogressed,ratherthanbeingconstructedpriortothestudy. Westarted with purposive sampling for initial response and because it requires the knowledge of insiders to locaterespondentsforthestudyweusedsnowballsamplingtoidentifyotherrespondents. Snowball sampling means that we asked the first purposively sampled respondents for additional relevant contactswhowouldbeabletoprovidedifferentorconfirmingperspectives[42,43]. Allrespondents employedbythemunicipalitywerefromtheDepartmentofHealth. In Case A, no prior ICIAs had been undertaken to prevent children getting affected by overweightandobesity. Amultinationalcorporationthatheadquarteredinthemunicipalityinitiated the JOGG-approach by convincing the municipality to implement the JOGG-approach. Case A was a medium sized urban municipality in the Netherlands of approximately 25,000 inhabitants. TheJOGG-approachwasimplementedinthewholemunicipality. InCaseBthemunicipalityinitiated the implementation of the JOGG-approach as a follow-up to an existing integrated approach to the prevention of lifestyle problems, including overweight. Case B was a somewhat larger urban municipality in the Netherlands with approximately 55,500 inhabitants. In this municipality, a community approach was chosen in which two neighbourhoods were involved of approximately 7400 people in total. In contrast to Case A, no other municipality that had implemented the JOGG-approachwasaffiliatedwiththeRPHSinCaseB. 2.3. DataCollection 2.3.1. Interviews Interviewswereconductedwithfiverespondentsineachcase(LvA).Respondentshadpositions withinthemunicipalityandwithintheRPHSatstrategic(oradministrative),tactical(ormanagerial), andoperational(orexecutive)level. Thedurationoftheinterviewsrangedfrom43minto67min. AspresentedinTable1,thetotalamountofhoursavailablefortheJOGG-approachwashigherin CaseBthanCaseA,respectively33.20hand10.45haweek. Table1.Individualcharacteristicsoftherespondents(n=10). WorkingTime Age Levelof Yearsof Respondent* Function Gender Organization forJOGG (Years) Education Service (Hours)** Programme A1 F 23 BS Municipality 2 8.00weekly manager A2 Policyadvisor F 28 MS RPHS 2 1.25weekly A3 Epidemiologist F 43 PhD RPHS 4 0.24weekly Health A4 promotion F 27 MS RPHS 4 0.96weekly professional A5 Alderman M 62 - Municipality 4 - Programme Municipality& B1 F 48 BS 2&13 20.00weekly manager RPHS B2 Policyadvisor F 55 MS Municipality 14 8.00weekly B3 Epidemiologist F 38 MS RPHS 12 4.00weekly Epidemiologist B4 &Policyadvisor F 37 MS RPHS 11 1.20weekly (RPHS-employee) B5 Alderman M 55 - Municipality 4 - *=LetterArepresentsCaseA,letterBrepresentsCaseB;**=Calculatedonthebasisof208workabledays (40-hourworkingweek).Abbreviationsused:BS=BachelorofScience,MS=MasterofScience,PhD=Doctor ofPhilosophy,RPHS=RegionalPublicHealthService. Int.J.Environ.Res.PublicHealth2016,13,390 5of19 Furthermore, the programme manager and policy advisor for Case B were on average ten yearsolderthantheonesforCaseA.RespondentfourofCaseAisindicatedas“HealthPromotion Professional”andrespondentfourofCaseBisindicatedas“RPHS-employee”(seeTable1). Priorto theinterview,allrespondentssignedaninformedconsent. Interviewswereconductedwithatopic list. This topic list was based on literature and the model of Preskill and Boyle [44]. Additionally, threeresearchers(LvA;AdK;MvK)developedthetopiclistcollaborativelyandcarriedoutapilot test. Thetopiclistevolvedovertimeastheinterviewprocesscontinued. Topicsfocusedontheroleof arespondentinevaluation,evaluationcompetence(knowledge,skills,andattitude),perceptionof theprogressofevaluation,resourcesforevaluation,organizationalstructureofthemunicipality,key personsinevaluation,andperceivedcollaboration. Firstly, programme managers and policy advisors of Case A and Case B were interviewed. InCaseA,LvAalsoattendedanintroductorymeetingattacticallevelwithanewaccountmanager employed by the JOGG-office. During this meeting the account manager asked questions of the programmanagerandpolicyadvisortooutlinethecurrentsituationoftheJOGG-approachinthe municipality. The responses from the interviews from Case A were supplemented with shared information. InsightwasgainedintoorganisationalstructureandthestatusoftheJOGG-approachin themunicipalities. Basedontheresultsofthefirstfourinterviews,complementaryinterviewswere performedwithotherstakeholdersatCaseAandCaseB.Astheinterviewprocessprogressed,the programmemanagerswereinterviewedagaintofurtherelaborateontheevaluationofactivitiesand responsibilityforevaluation. Aftereachinterview,LvAwroteareporttorecordthefirstimpressions andabriefdescriptionoftherespondentandinterviewlocation,thecourseoftheinterview,summary, andkeywords. Therecordingsweretranscribedverbatim. Respondentsreceivedasummaryofthe interviewinordertocheckthecredibilityofthefindings(membercheck)[45]. 2.3.2. Observations Observations were conducted during meetings at strategic programme level (e.g., steering committee)andoperationalprogrammelevel(e.g.,workinggroups)ofboththeJOGG-approaches. In Case A, the researcher (LvA) conducted an observation at operational project level. In Case B two observations were conducted at the strategic project level and one at the operational project level. Notesregardingthesetting,actions,mutualpowerrelationships,modeofinteraction,content, decision-making, and non-verbal cues of all stakeholders were made, and the extent to which stakeholders participated during the meetings. Afterwards, LvA continued informal discussions with stakeholders where possible (e.g., during a car ride). In this way, she was able to strengthen therelationshipsshehadestablishedinearlierinterviewsandwasabletoaskquestionsinamore informalway. Aftereachdayofobservation,LvAmadecomprehensivefieldnotesofherobservations andinformaldiscussions. Thesedetailednotesgeneratedinsightandbetterunderstandingofthedata collectedduringtheinterviews. 2.3.3. DocumentAnalysis Allinterviewedrespondentswereaskedtosharedocumentsoutliningtheprojectstructureand currentsituationofthe(evaluationofthe)JOGG-approachanddocumentscontaininginformation abouttheembeddingoftheJOGG-approachinpolitics. Avarietyofdocumentswerereviewedin eachcase: programmeplans,minutesofmeetings(atstrategicandoperationallevel),overviewsof resolutionsoftheMunicipalBoard,MemorandumsofPublicHealth,newsletters,andorganization charts. Theinformationextractedfromthedocumentationwasusedintwoways: (1)asinputfor interviewsortohighlightsituationsthatneededtobeobserved;and(2)tocross-validatedatagathered duringtheinterviewsandobservations[46]. Int.J.Environ.Res.PublicHealth2016,13,390 6of19 2.4. DataAnalysis Conceptsfromthemulti-disciplinarymodelofevaluationcapacitybuilding(ECB)ofPreskill andBoyle[44]wereusedinthedevelopmentofthecodingscheme. Althoughourresearchdoesnot focusonECBitself,exploringthebarrierstoandfacilitatorsofperformingevaluationmayincrease insightintowhatisneededtobuildevaluationcapacityamongprofessionalsinICIAs. Thisconceptual frameworksuggestsmultipleindividualandorganizationalresourcesthatshouldbeinplaceinorder tobuildlocalevaluationcapacityandestablishasustainableevaluationpractice[44]. Whenthese resourcesarenotinplaceasustainableevaluationpracticeisnotfeasible. Theinterviewtranscriptsandobservationswereanalysedbytworesearchers(LvA,AdK)using thematiccontentanalysis[47]. Thetranscriptswerereadthroughseveraltimes. Thetextsweredivided intofragments,andcodes(labels)wereassignedtothesefragments(opencoding). Thesecodeswere allorganizedintoamindmap. ThepreliminaryconclusionswerethoroughlydiscussedwithMvK. Thelastphaseoftheanalysiswasselectivecoding: theessenceofwhateachthemewasaboutwas identified,searchedforrelationsthroughconstantcomparisonacrosscasesandanalysedvariation withinandbetweencases. Finally,thedifferentthemeswerefittedintothebroaderpictureoftheentire data. Duringtheentireanalyticalprocess,memoswereusedtorecordthoughtsabouttheresultsand todistinguishbetweentheresearchers’interpretationsandrespondents’ownviews. Toensurevalidity andreliabilitythoughtsandanalysiswerediscussedwithintheresearchteam. 3. Results 3.1. CaseDescription Both cases had adopted the JOGG-approach in the same year, 2012, but with different initiatorsandpointofdeparture. InCaseA,amultinationalcorporationthatheadquarteredinthe municipalitybroughttheJOGG-approachtotheattentionofthealderman,whointurnsupportedthe implementationinordertomaintainagoodrelationshipwiththisprivatecompany. Nopreparations hadbeenundertakenpriortotheJOGG-approach: themunicipalitystartedfromscratch,asreflected bytheprogrammemanager’sperceptionofherfunction: “Sometimes(Ihave)thefeelingof: ‘wheretostart?’ and(itfeelslike)dropletsintheocean”(A1). InCaseB,theinitiatoroftheJOGG-approachwasthemunicipality. Aprogrammesimilartothe JOGG-approach,calledprogrammeXinthisstudy,hadalreadybeenimplemented. ProgrammeX wasanintegratedapproachtothepreventionoflifestyleproblems,includingoverweight,andwas accompaniedbyfinancialandsubstantivesupport. Therefore,acertainknowledgeandexperience basewasalreadyestablished. AccordingtotheprogrammemanagertheJOGG-approachwasusedto amplifythepreviousprogramme. Inbothcasesasteeringgroupandmultipleworkinggroupswereestablished. Atstrategiclevel bothJOGG-approacheswerecoordinatedbyanaldermanandthedepartmentheadofthehealthsector ofthemunicipality. Inbothcasesa“steeringgroup”functionedmoreontheoperationallevelthanon thestrategiclevel. ArespondentfromCaseAdescribedthesteeringgroupashavingalackof: “... realvision,policy,andmakingdecisions(atstrategiclevel)”(A1). MorepeoplewerepartoftheprojectstructureinCaseBthaninCaseA.ForexampleinCaseA eight people were part of the working group “Interventions” and in Case B there were twelve. ThesteeringgroupofCaseAhadsevenmembersandinCaseBeleven. CaseAfocuseditsprogramme structurearound“Interventions”and“Communication”.AndCaseBfocusseditsprogrammestructure around“Interventions”,“Careandprevention”,and“Evaluation”. TheinterviewedprofessionalsinCaseAwereyoungerandhadnotbeenintheemploymentof theirorganisationsforaslongasthoseinCaseBandwerelessexperiencedwithevaluatingpublic healthinitiatives. Moreover,workingtime(hours)fortheJOGG-approachwashigherinCaseBandin Int.J.Environ.Res.PublicHealth2016,13,390 7of19 thiscasemoreprofessionalsfrominandoutsidethelocalgovernmentandatdifferentlevels(strategic, tacticalandoperational)wereinvolved. Inbothcases,noevaluationteamwasestablishedeventhoughthiswasrecommendedinthe JOGG evaluation manual and in the first (of four) evaluation training session of the JOGG-office. The programme manager and the epidemiologist in both cases attended this first training session. TheprogrammemanagerofCaseBexplainedtheydidnotestablishaseparateevaluationteam,but madeevaluationpartofexistingworkinggroups.Itappearsthiswasanon-formalisedagreementsince allrespondentsofCaseBwereinvolvedinevaluationandhadtheirownstrategicandoperational tasksandroles. TheprogrammemanagerofCaseAsaidtheydoubtedtheneedforsuchateam,saida clearmandateforsuchateamhadnotbeengivenbythesteeringgroup,andtherewerenoresources forevaluation. 3.2. EvaluationBarriersandFacilitators IntheupcomingparagraphswedescribeourmergedresultsfromCaseAandCaseB.Perceived barriers to and facilitators of evaluation in each case are presented using direct quotations from interviews (indicated by italic letters and enclosed in quotation marks), field notes, observations, and documents (enclosed in quotation marks). A total of six themes derived as key-issues after data-collection: motivatingfactorstoevaluate,perceivedfeasibilityofevaluation,knowledgeand attitudesofevaluation,communicationandinvolvementwithevaluation,evaluationresources,and supportfromdecision-makers. AnoverviewoftheresultscanbefoundinTable2. Table2.BarrierstoandfacilitatorsoftheevaluationoftheJOGG-approach. Barriers(´)/ Themes Subthemes Examples Facilitators(+) Theneedforanevaluationmotive Anevaluationexpertwhoprovidesexpertiseandsupportto Persontomotivate + Motivating starttheevaluationprocess evaluation factorsto ´ Lackofapersonwhostimulatesperformanceofevaluation evaluate Demandfor + Externalfunderoraldermantoaskforevaluationresults evaluation Theprogrammemanagerdoesnotgiveacommandtostart ´ theevaluationprocess Evaluationfeasibility Assumptionson + Existenceofarealisticperceptionofevaluation Perceived feasibilityof feasibilityof evaluation ´ Negativeperceptionsonfeasibilityofevaluationaspresented evaluation intheoryandevaluationmodels Capabilitiesof + Trustininterpretationoftasks thoseinvolved Lackoftrustincapabilitiesofthosethatshouldbeinvolvedin ´ theevaluation:programmemanagerandepidemiologist. Positiveattitude + Evaluationisregardedasimportant towardsevaluation Knowledgeand ´ DoubtaboutpossibilitiestoshoweffectsofICIA attitudeson evaluation Knowledgeon Availabilityofapersonwithsufficientknowledgeonwhatthe + (processof)programmeevaluationimpliesandhowto evaluation conductsuchanevaluation Lackofevaluationknowledge(i.e.,oralinformalprocess ´ evaluationhasnoscientificvalue) Perceptionofown + Highself-efficacytoconductevaluation capabilities ´ Negativeperceptionofone’sowncapabilitiestoevaluate Int.J.Environ.Res.PublicHealth2016,13,390 8of19 Table2.Cont. Barriers(´)/ Themes Subthemes Examples Facilitators(+) Evaluationfeasibility Financialresources + Allocatedfinancialresourcesforevaluationprocess ´ Limitedresourcestohirepersonnel,resultinginlimitedtime Evaluation resources Time + Allocatedhoursforinvolvementofepidemiologistor evaluationexpert Lackofpersonnelfordatacollection;Lackoftimefor ´ evaluationeducation Availabilityof Availabilityofgenericsuitableevaluationinstruments evaluation + (i.e.,questionnaires,logicmodel) instruments ´ Non-functioningorincompleteevaluationinstruments Commitmentofevaluationstakeholders Regularandhighdegreeofcommunicationbetween Communicationon + Communication programmemanagerandepidemiologistonevaluation evaluation andinvolvement Lowdegreeofcommunicationonevaluationbetween withevaluation ´ programmemanagerandepidemiologist;Lowdegreeof communicationbetweenprogrammemanagerandalderman Activeinvolvementofstakeholdershelpstoseetheadded Involvementin + valueofevaluation evaluation Involvementofmembersoftheprogrammestructureat ´ strategicaswellastacticalandoperationallevel Evaluationisconsideredimportantbyalderman Politicalsupport + andcitycouncil (tactical) Supportform Competingthemesthatreduceattentionandmakefewer decision-makers ´ resourcesavailable;Politiciansonlyinterestedinlong-term atmultiplelevels goalsandnotinmid-termorprocessevaluations Apro-activeattitudeofdepartmentmanagementtogenerate Managerial + resourcesandsupport;aclearpolicyvisionofRPHS support(strategic) supportivetotheapproachandevaluation Atimeconsumingpolicyprocesstogenerate ´ extrafinancialresources Supportfrom + Stakeholdershaveacommongoal implementers Limitedinterestinevaluationfromthosewho (operational) ´ implementtheICIA 3.2.1. MotivatingFactorstoEvaluate Analysis of data showed that a trigger or motivating reason could give rise to an evaluation (CaseB).Ontheotherhand,thelackofitcouldimpedeevaluation(CaseA).Allintervieweesasked saidaspecificpersonwasneededtomotivateevaluation. Inparticulartheprogrammemanagerwas seenashavinganidealpositiontomobiliseandmotivateprofessionals: “Theprogrammemanagerisanenthusiasticpersonwhocantransferherenergyproperly. Sothat worksverywell”(B4). The programme manager mentioned (Case B) also seemed to understand her role to trigger evaluationprocessessincesheaskedforevaluation-relatedactivities: “Myrole(asprogrammemanager)istoquestion(..) them(thestakeholders)aboutthebenefitof theiractivities. Sointhatwaywetrytocollectdata(aboutevaluation)”(B1). InCaseB,thealdermanalsoprovidedsuchatriggerbyaskingforaneffectevaluation. Moreover, theprogrammemanagerandpolicyadvisorinCaseBsaidanexternalpublicfunderofprogrammeX providedanadditionaltriggerbyrequestingaprocessevaluationinafinalreportforaccountability reasons. AlsotheevaluationexpertwhowasmadeavailabletomunicipalitiesbytheJOGG-office Int.J.Environ.Res.PublicHealth2016,13,390 9of19 was a catalyst in the design and execution of evaluation and helped others to understand the evaluationprocess. “( ... )itwasgoodtohavetheevaluationexpertfromtheJOGG-officevisitusanddiscussthe evaluationpossibilitiesandeffectstobeshownatlocallevel(...),thisworksbetterthanthetraining sessionsalone,whicharemoregeneral”(B1). However,thisprogrammemanagerwouldhavelikedmoredirectivesonevaluationfromthe JOGG-office. In contrast, interviewees in Case A said no-one triggered evaluation. To stimulate evaluation,thepolicyadvisorandhealthpromotionprofessionalfromCaseAthereforesuggestedthat anepidemiologistfromtheRPHS,theJOGG-office,aprogrammemanagerorthealdermanshouldask forevaluation. Theevaluationtrainingsessionswerenotperceivedasstimulatingtheprogramme managerofCaseAtocommenceevaluationplanning: “(... )thetrainingsessionsweretooformal,extensive,detailorientedandexaggerated”(A1). The given information was too overwhelming for her and did not increase her motivation to studytheprocessofevaluationletalonestimulatehertodesignandperformtheevaluationofthe JOGG-approach. 3.2.2. PerceivedFeasibilityofEvaluation Interviewees in both cases perceived evaluation of the JOGG-approach as unfeasible. Theyassumedthatasoundandproperevaluationcouldnotbeperformedwiththeirlimitedbudget, time and capabilities. For example, the programme manager of Case A and policy advisor of CaseBsaid: “There is so much to it (evaluation) if you want to do it well. A lot of time, money (..), and expertise”(A1). “Ifyouwanttodoit(evaluation)properly,youhavetomakeabiginvestment”(B2). Mostintervieweesaddedthattheevaluationaspresentedintheevaluationtrainingandinthe evaluationmanualoftheJOGG-officerepresentedan“ideal”evaluationwhichwasnotencounteredin practiceorwasevenincompatiblewithpractice. Someintervieweesstruggledwiththetranslationof theevaluationmanualforuseintheirownsituation. BoththeepidemiologistandtheRPHS-employee of Case B explained that evaluation of the JOGG-approach was not systematic, but “emerged” duringimplementation: “Idonoticethatpracticeismorestubborn(thantheidealsituation),sothenyou’llhavetolook furtherandsearchforyourownpathway”(B3). “Youwanttowritethingsdownasspecificallyaspossibleinsuchan(evaluation)plan. However, JOGGisalsocharacterizedby(..) abottom-upapproachandsettinggoalstogether. Andsometimes Iperceivethatasquiteatension”(B4). 3.2.3. KnowledgeofandAttitudestoEvaluation Inbothcasestheprogrammemanagersindicatedtheylackedsufficientknowledgeofevaluation. The epidemiologists were believed to be sufficiently equipped to design and perform evaluation. However,thepolicyadvisorinCaseAsaidthatsomeepidemiologistswhowerebelievedtobecapable ofdesigningandconductinganevaluationwereinrealitynotabletodoso. Thismightbeexplained bythefactthatmostinterviewees,includingtheepidemiologists,didnotunderstandthedifference betweenevaluationandresearch,theyusedthetermsinterchangeably,hencemakingevaluationbigger andmoredifficultthannecessary. Forinstance,theprogrammemanagerandpolicyadvisorofCaseA frequentlyusedthetermsevaluationandresearchinterchangeablyandreferredtotheworkinggroup Int.J.Environ.Res.PublicHealth2016,13,390 10of19 “Research”whentheintervieweraskedaboutevaluation. Askingabouthercompetencesinevaluation, theprogrammemanagerofCaseAreplied: “Ididsomeresearchduringmystudybutnotlikeayouthmonitororsomething. Iwouldnotknow whatquestionstoaskorwhensucharesearchisrepresentativeorreliable(... )”(A1). Moreover,intervieweesinCaseAdidnotperceiveaprocessevaluationintheformofanoral communicationofanactivitytodiscusstheprogressandpossibleareasofimprovement,aspartof theprogrammeevaluation,becauseitwasnotformalenough. InCaseB,theprogrammemanager, epidemiologist,andRPHS-employeeknewthatprocessevaluationwasimportanttooptimizethe approach,buttheirattitudestotheprocessevaluationwerelesspositive: “ithasnoscientificvalue”(B2)and“itisnotsomethingofficial”(B4). Theepidemiologistfeltthatsheandtheotherprofessionalswerenotfullyequippedtoconducta processevaluation. Additionally,theprogrammemanagerandRPHS-employeeofCaseBshowed theywereawareofdifferencesinevaluationbetweendifferentinitiatives,theysaidevaluationcould eitherbeextensiveorsuperficialdependingonneedsandfunding. Incontrasttothedifferentpurposesofevaluationandresearchasdiscussedintheintroduction tothispapertheprogrammemanager,policyadvisor,andhealthpromotionprofessionalofCaseA allmentionedthatevaluationfindingscouldbeusedto“prove”something. Theseperceptionson evaluation can limit the use of evaluation for programme optimization and diminish the value of evaluationatstrategicandtacticallevels. Regardingrespondents’attitudestowardsevaluation,somerespondentsquestionedwhetherthe effectsobservedcouldbeallocatedtotheJOGG-approach. Thepolicyadvisorandepidemiologistof CaseB,forexample,explaineditwouldbedifficulttoproduce“hard”dataabouttherelationship between the JOGG-approach and effects. Furthermore, several respondents cited that evaluation requiredalotofinvestmentofresources. Onerespondentcommented: “Anditisthequestionwhetherit(theevaluation)isworththeinvestment”(B2). Evaluationingeneralwasconsideredimportantbythemajorityoftherespondents: “Whenyouevaluate,youknowwhatworksandwhatdoesnot(..). Andyoucankeeppartnerson board,showresults,(create)apositiveatmosphere,andattractotherparties”(A1). “Itisimportanttoseewhetherwhatyoudoachieveseffects”(B2). 3.2.4. EvaluationResources Datarevealedthatlackoftimeandallocatedfinancialresourcesprohibitstheevaluationofthe JOGG-approach(CaseA).OnmultipleoccasionstheprogrammemanagerofCaseAstatedthatlimited financialresourceswereavailabletoadministerquestionnaires,tohirepersonnelfordatacollection andanalysis,andtosetupaproperevaluation. Inthismunicipalityabudgetforevaluationofthe JOGG-approachwasnotgenerated. SincetheRPHSmonitorshealthandbehaviourofchildreninthe municipalityitwasdecidedtousethisdatatoshowresults,despitethelimitationsofthisdata. In CaseB,ontheotherhand,allocatedtimeandfinancialresourceswerebelievedtohavestimulatedthe evaluationprocess: “Within the entire process we have allocated time and money in order to conduct a proper evaluation”(B2). Interviewees in Case A reported that the annual fee of the JOGG-approach was paid by the multinationalheadquarteredinthemunicipality,whileinCaseB,theannualfeewasfundedbythe subsidyforprogrammeXandreservedresourcesfromlocalhealthpolicy. Concerningtime(personnel resources), hourswereallocatedforanepidemiologistinCaseB,whereasnoadditionalhoursfor
Description: