P.Re.Val.E.: outcome research program for the evaluation of health care quality in Lazio, Italy.

0.31 MB·English

by  Fusco, Danilo

Fuscoetal.BMCHealthServicesResearch2012,12:25
http://www.biomedcentral.com/1472-6963/12/25
RESEARCH ARTICLE Open Access
P.Re.Val.E.: outcome research program for the
evaluation of health care quality in Lazio, Italy
Danilo Fusco1*, Anna P Barone1, Chiara Sorge1, Mariangela D’Ovidio1, Massimo Stafoggia1, Adele Lallo1,
Marina Davoli1 and Carlo A Perucci2
Abstract
Background: P.Re.Val.E. is the most comprehensive comparative evaluation program of healthcare outcomes in
Lazio, an Italian region, and the first Italian study to make health provider performance data available to the public.
The aim of this study is to describe the P.Re.Val.E. and the impact of releasing performance data to the public.
Methods: P.Re.Val.E. included 54 outcome/process indicators encompassing many different clinical areas. Crude
and adjusted rates were estimated for the 2006-2009 period. Multivariate regression models and direct
standardization procedures were used to control for potential confounding due to individual characteristics.
Variable life-adjusted display charts were developed, and 2008-2009 results were compared with those from 2006-
2007.
Results: Results of 54 outcome indicators were published online at http://www.epidemiologia.lazio.it/prevale10/
index.php.
Public disclosure of the indicators’ results caused mixed reactions but finally promoted discussion and refinement
of some indicators.
Based on the P.Re.Val.E. experience, the Italian National Agency for Regional Health Services has launched a
National Outcome Program aimed at systematically comparing outcomes in hospitals and local health units in Italy.
Conclusions: P.Re.Val.E. highlighted aspects of patient care that merit further investigation and monitoring to
improve healthcare services and equity.
Background The Mattoni-Outcome Project, funded by the Italian
Overthelasttwodecades,therehasbeenincreasinginter- Ministry of Health[13], and the subsequent Progr.Es.Si.
est in the development and implementation of outcome Project, have beenthe mainnational experiences in this
and process indicators. Such indicators encourage field.TheMattoni-OutcomeProjectmethodologieswere
accountabilityandimprovements inthequalityofhealth the starting point for the Regional Outcome Evaluation
careservices;theyalsoguideaccreditationandhealthcare Program,calledP.Re.Val.E.[14,15],whichwasconducted
planninginterventions[1-6].Publicandprivateorganiza- intheLazioregionofItaly(5,493,308residents[16]).P.
tions,aswellasresearchprojects,haveuseddifferentindi- Re.Val.E.isthemostcomprehensivecomparativeevalua-
cators for comparative evaluation of the performance of tionofregionalhealthcareoutcomesandtheonlyItalian
healthcareprovidersandprofessionals.Somehavereleased studytopubliclydiscloseperformancedata.
theirresultstothepublicintheformofweb-basedreports The objectives of P.Re.Val.E. were to define and evalu-
thatcomparehospitalquality[7-13]. ate outcome/process indicators in order to:
In Italy, national and regional outcome research pro-
grams have been conducted, but there has been no sys- (cid:129) compare the outcomes of health care provided by
tematic comparison of outcomes at the national level. different hospitals or in different geographical areas
for purposes such as accreditation, remuneration,
*Correspondence:[email protected] and enhancing citizen empowerment;
1DepartmentofEpidemiology,RegionalHealthService-LazioRegion,via
SantaCostanza53,Rome,00198,Italy
Fulllistofauthorinformationisavailableattheendofthearticle
©2012Fuscoetal;licenseeBioMedCentralLtd.ThisisanOpenAccessarticledistributedunderthetermsoftheCreativeCommons
AttributionLicense(http://creativecommons.org/licenses/by/2.0),whichpermitsunrestricteduse,distribution,andreproductionin
anymedium,providedtheoriginalworkisproperlycited.
Fuscoetal.BMCHealthServicesResearch2012,12:25 Page2of10
http://www.biomedcentral.com/1472-6963/12/25
(cid:129) compare population subgroups (e.g. socioeconomic TheHISdatabasewastheprimarysourceforcaseselec-
subgroups), especially for evaluating and promoting tion and outcomes/comorbidities data. We also used
equitable service provision; information from the EIS database to better identify
(cid:129) identify the minimum volume of activity associated comorbiditiesandtoestimatethetimetodeathortimeto
with the best treatment outcomes; surgery after arrival at the hospital (i.e. admission to the
(cid:129) promote internal and external auditing; Emergency Department or to a hospital ward). Deaths
(cid:129) identify critical areas in which to implement pro- duringthestudyperiodwereidentifiedusingtheHIS,the
grams that improve health care quality; EIS,andthe MIS.TheRADRdatabasewasused to iden-
(cid:129) monitor trends in health care quality over time. tifyadmissiontoarehabilitationcenterfollowinghospital
admissionforstroke.Finally,we usedtheHDISdatabase
In order to highlightdifferences in the performance of for more accurate identification of new births, primary
localcommissioningauthorities,P.Re.Val.Eevaluatedout- cesarean deliveries, and risk factors for cesarean section
comesaccordingtopatientresidence.Importantly,direct whicharenotincludedintheHIS.
standardizationmethods[17-20]ratherthanindirectstan- HIS records were linked with EIS, MIS, RADR, and
dardization methods [5,7,10,20] were used for the com- HDIS records using deterministic record-linkage. To
parativeevaluationofoutcomes.Theaimofthisreportis ensuremaximumcoverageofthepopulationwhileavoid-
todescribetheP.Re.Val.E.program,includinginformation ing double counting, the linkage method uses a unique
sources, statistical methodologies, results for 2006-2009 patient identifier deriving from information on persons’
(published online at http://www.epidemiologia.lazio.it/ names, date and place of birth and gender, according to
vislazio/vis_index.php,andthe updated Italianversionat Italianprivacylegislation.
http://www.epidemiologia.lazio.it/prevale10/index.php),
and the impact of releasing performance data to the Study population
public. P.Re.Val.E.analyzedhospitaldischargesintheLazioregion
inthe2006-2009period[14].Mostdatawereexpressedas
Methods ratios in which the numerator represents the number of
Study design and data sources treatments/interventions provided or the number of
Wedefined54performanceindicatorsindifferentclinical patientswitha givenoutcome (i.e.,short-termmortality,
areasthatincludedtreatmentforcardiac,cerebrovascular, hospitalization for specific conditions, etc.) and the
orthopedic, obstetric,respiratory,anddigestivedisorders denominator represents the group of patients at risk. In
[14]. Some were health outcome indicators, such as 30- othercases,theindicatorsweredefinedassurvival/waiting
daymortalityafteranepisodeofmyocardialinfarctionor time(e.g.,thewaitforsurgeryafterhipfracture).
30-day complications aftercholecystectomy;otherswere The analyses were performed, two years over two
processindicatorsforwhichanassociationwithimproved years, by area of residence regardless of the hospital at
health outcomes has been already proven, such as inter- which the patient was treated and by hospital.
ventionwithin48hoursofhospitaladmissionforhipfrac-
ture in the elderly. Finally, hospitalization rates for Definition and attribution of outcome
pathologies generally treated out of the hospital, such as Theoutcomemeasureswereasfollows:30-daymortality,
diabetes,asthma,andinfluenza,wereconsideredanindi- short-term re-hospitalization, hospitalizationfor specific
cationoffailureinprimarycare[21]. conditions,surgicalprocedures,short-termcomplications
Most indicators were selected based on their previous ofspecificinterventions,andwaitingtimes.Theoutcomes
use in international and national studies [7-19,21-24], understudywereattributedtothefirstemergencydepart-
whileotherindicatorsweredevelopedinordertodescribe ment at which the patient was treated(firstaccess)orto
particular components of care or clinical pathways. A thefirstadmissionhospitalandtotheareaofresidence.
rationaleandadetailedoperativeprotocolwereelaborated
for each indicator according to a standard outline [14]. Coexisting medical conditions
The indicators were defined using information collected Chroniccomorbiditiesand/orseveritycharacteristicsthat
from regional health information systems covering the were potentially associated with the outcomes under
wholeLaziopopulation:theHospitalInformationSystem study were chosen using information in the literature
(HIS),theEmergencyInformationSystem(EIS),theMor- [7-9,17,18,21-23,25-30] and in the Mattoni-Outcome
tality Information System (MIS), the Report Admission- project[13].Thepotentialriskfactorswereidentifiedon
Discharge for Rehabilitation (RADR), and the Hospital the basis of ICD-9-CM codes registered either during
DeliveriesInformationSystem(HDIS).Fordetailedinfor- hospitalizationfortheconditionunderstudy(indexhos-
mation about each,please seehttp://www.epidemiologia. pitalization)orinprevioushospitaloremergencydepart-
lazio.it/vislazio_en/fonti.php. ment admissions during the previous two years. Acute
Fuscoetal.BMCHealthServicesResearch2012,12:25 Page3of10
http://www.biomedcentral.com/1472-6963/12/25
eventsoccurringduringtheindexhospitalization,which usedtheclustersandwich("robust”)variance-covariance
could be complications of care/treatments (i.e. on the estimators,relaxingtheusualrequirementthattheobser-
causal pathway between exposure and outcome), were vationsareindependent.Accordingtothismethodology,
notincluded. the observations are independent across groups but not
necessarilywithingroups.
Statistical analysis The adjusted RR estimated for each hospital/area of
Statisticalanalyseswere performedbyfirstadmissionor residence, the adjusted risk or median waiting time, and
access hospital as well as by area of residence. Since the corresponding p-value were reported on-line in tab-
patient characteristics, such as age, sex, severity of dis- ular and graphical forms.
ease, and/or chronic comorbidities, could be heteroge- For each indicator, trend analyses and comparisons of
neously distributed across the hospitals/areas of the 2008-2009 data versus the 2006-2007 data were
residence, risk-adjustment methods were applied [31]. developed by hospital and area of residence.
The riskadjustmentprocedure involved construction of Variable Life-Adjusted Display (VLAD) charts [34,35]
a severity measure (“a priori” risk) that was specific for were constructed to identify the principal increases/
the study population and the use of this measure to decreases and trend reversals in the cumulative sums of
obtain “adjusted” outcome measures for comparison the differences between the number of events observed
betweenhospitals/areas.Theseveritymeasurewascalcu- (deaths, rehospitalizations, reoperations, complications,
lated by analyzing the multivariate relationship between etc.) and the number expected based on the predictive
possibleoutcomepredictorsandtheoutcomeconsidered model. Two tests of statistical significance were per-
byapplicationofmultivariateregressionmodels(predic- formed, the first to determine whether during the study
tive models) including: 1) a prioririsk factors (age, sex, period, the number of events observed in one month
and/orseverity of the conditionbeinginvestigated); and (from the second day of the previous month to the first
2) factors selected by a bootstrap stepwise procedure. day of the following month) was significantly different
Stepwise analysis was performed with 500 replicated from the number of expected events and the second to
samples of the original data and significance thresholds determine whether the number of observed events
of0.10and0.05forentryandremoval,respectively. between2trend-reversalpointswassignificantlydifferent
Only risk factors (pre-existing chronic conditions, etc.) from the number of expected events. Both significance
selected in at least 50% of the runs were included in the tests were performed using the ratio of observed to
final models expectedeventsassumingaPoissondistribution.
We used logistic regression models for dichotomous Finally, since higher activity volumes of the hospitals
outcome variables and survival models for outcomes areoftenassociatedwithbetteroutcomes,wealsoinves-
expressedintermsofsurvivaltimes. Inordertoestimate tigatedthoserelationships(displayedasscatterplots).
adjusted group-specific (hospital/area of residence) log The level of statistical significance was set at 5% (p <
oddsofoutcome,logisticregressionmodelswithnointer- 0.05), and all analyses were performed using SAS Ver-
cept and centred covariates were applied for each out- sion 8.2 [36].
come. Adjusted risks were obtained for each group by
back-transformingparameterestimateswiththefollowing Preliminary disclosure of performance data to clinicians
formulas: and providers
Before public release of data, we shared the P.Re.Val.E
Adjrisk=[exp(estimate)/(1+exp(estimate))]*k
methods and results with different groups of clinicians
andproviders,topromotediscussionandencouragecon-
where k is a correction coefficient introduced to take
tributionsandcriticalassessments.
into account the nonlinear nature of the logistic model.
K is calculated as follows:
Results
actual number of events Wecalculated54indicators,9ofwhich wereprevention
k=
(cid:2)m qualityindicators,thatcoveredalmost40%ofallhospital
p ∗n
j j admissions in the Lazio Region during the 2008-2009
j=1
period. The results obtained for all indicators, available
where p are the adjusted risks, n is the group size, online at http://www.epidemiologia.lazio.it/prevale10/
j j
and m is the number of groups. index.php,showedagreatheterogeneityofthehealthcare
Thisapproachallowed comparisonoftheoutcomefor qualityintheLazioregion.
agivenfacilityorareaofresidencewiththatofthewhole Public disclosure of the indicators’ results caused
study population and with each of the other facilities/ mixed reactions but finally promoted discussion and
areas[18,32,33].Inordertoassessforclustereffects,we refinement of some indicators, e.g., for 30-day mortality
Fuscoetal.BMCHealthServicesResearch2012,12:25 Page4of10
http://www.biomedcentral.com/1472-6963/12/25
after aortic aneurysm. These meetings led to new esti- Theadjustedmortalityratesrangedfrom6.7%to19.2%in
mates for some indicators and stimulated audit activities different hospitals, and from8.8% to 13.7% for different
among clinicians and healthcare organizations. areasofresidence(Tables2and3).Intheprevious2-year
In 2008, Agency for Public Health of Lazio designed a periodtherewere17436AMIepisodes,themeanmortality
clinical pathway for elderly patients with hip fracture. ratewas12.2%,withhighvariabilityamongthedifferent
The clinical pathway was tested in five selected hospitals facilitiesandareas.Ingeneral,theadjustedmortalityrates
before the implementation in all Lazio hospitals. showedasmalldecreaseovertimeformostfacilitiesand
In 2009, a regional health service regulation went into areas(datanotshown).AlthoughtheVLADchartswere
effect that required Lazio hospitals to adopt the clinical developedforeachfacilityandarea,theresultsfromtwo
pathway for elderly patients with hip fracture and that largehospitalsthatshowedverydifferenttimetrendsfor
introducedacompensationsystemforhospitalsbasedon 30-daymortalityafterAMIadmissionwerereportedonly.
qualityofhealthcare(asinapay-for-performancemodel). In2008-2009,PresidioOspedalieroNord,Latina,showeda
The DRG reimbursement rate for Lazio providers was greaternumberofobserveddeathsafterAMIadmission
linkedtohospitalperformance.Infact,from2009on,the than expected, with a little decrease in November 2009
fullDRGratehasonlybeenpaidforpatientswhounder- (Figure1),whereasSanFilippoNeri,Rome,generallyhad
went surgicaltreatmentwithin48hours after admission, fewerobservedeventsthanexpectedones(Figure2).
while rates for interventions performed after 48 hours
werereducedproportionallybasedonthetimetosurgery. Discussion and Conclusions
BasedontheP.Re.Val.E.experience,theItalianNational Comparativeevaluationofhospitalperformanceisause-
Agency for Regional Health Services has launched a ful tool for improving healthcare quality [1,4]. The U.S.
NationalOutcomeProgramaimedatsystematicallycom- experiencehasshownthatpublicdisclosureofcompara-
paring outcomes in hospitals and local health units in tiveevaluationresultsshouldbemanagedasonecompo-
Italy. nent of an integrated quality improvement strategy, and
As anexample, results pertaining the 30-day mortality thatthepublicreleaseofperformancedataismosteffec-
rateafterhospitaladmissionforacute myocardialinfarc- tiveattheleveloftheproviderorganization[3,4,37].Reg-
tion(AMI)inLazioregionwereshown[fortheoperative ular feedback seems to increase the accountability of
protocol see: additional File 1, Appendix: operative providers,whicharesensitivetopublicimageandpoten-
protocol]. tiallegalrisks;itcanalsospurqualityimprovementactiv-
Therewere16,682AMIepisodesin2008-2009witha ities in health care organizations, especially when
meanmortalityrateof11.1%(men:8.9%;women:15.2%). underperformingareasareidentified[38].However,pro-
Theprobabilityofdeathwas2timeshigherwhenchronic viders that are identified as poor performers are more
diseases(liver,pancreas,intestine)intheindexadmission, likely to question the validity of the data, particularly
cancerandothercardiacoperationswerepresent(Table1). whentheresultsarefirstreleased[39].
Table 1Acute myocardial infarction: mortality within 30 daysofhospital admission, predictive model
RiskFactors N CrudeOR AdjustedOR p
Age(years) - 1.08 1.08 0.000
Gender(FemalesvsMales) 5767 1.84 1.05 0.419
Cancer 810 2.52 2.05 0.000
Hypertention 3072 1.52 0.85 0.024
Previousmyocardialinfarction 2481 0.96 0.73 0.000
Cardiomyopathy(indexadmission) 259 0.82 0.63 0.040
Cardiomyopathy 218 1.88 1.14 0.505
Heartfailure 1117 2.77 1.62 0.000
Otherheartconditions(indexadmission) 367 1.06 0.60 0.006
Otherheartconditions 228 1.83 1.08 0.688
Cerebrovasculardisease 1122 2.43 1.47 0.000
Vasculardisease 679 2.23 1.57 0.000
Chronicrenaldisease 876 2.82 1.66 0.000
Otherchronicdisease(liver,pancreas,intestine)(indexadmission) 105 2.52 2.33 0.001
Otherchronicdisease(liver,pancreas,intestine) 209 1.98 1.35 0.135
Previouscoronaryangioplasty 1417 0.57 0.65 0.000
Othercardiacinterventions 127 1.69 2.06 0.006
Fuscoetal.BMCHealthServicesResearch2012,12:25 Page5of10
http://www.biomedcentral.com/1472-6963/12/25
Table 2Acute myocardial infarction: mortality within 30 daysofhospital admission, by health carefacility
Hospital Location N Cruderate×100 Adjustedrate×100 AdjustedRR p
OSP.S.CAMILLODELELLIS RIETI 509 12.57 12.23 1.10 0.461
OSP.S.SPIRITO ROME 574 10.28 8.43 0.76 0.049
OSP.ANZIO-NETTUNO ANZIO 351 9.69 9.11 0.82 0.269
OSP.ALBANO-GENZANO ALBANO 361 13.02 12.75 1.15 0.366
OSP.S.PAOLO CIVITAVECCHIA 298 14.77 13.86 1.25 0.162
OSP.PARODIDELFINO COLLEFERRO 320 8.75 8.23 0.74 0.129
OSP.S.SEBASTIANO FRASCATI 250 12.80 11.99 1.08 0.675
OSP.S.GIOVANNIEVANG TIVOLI 200 19.50 19.17 1.73 0.001
OSP.GRASSI ROME 648 9.72 9.17 0.83 0.152
OSP.S.EUGENIO ROME 485 13.81 12.25 1.10 0.454
OSP.S.PIETROF.B.F. ROME 232 12.93 14.02 1.26 0.216
OSP.VANNINI ROME 706 10.62 10.89 0.98 0.874
CCS.ANNA POMEZIA 179 12.29 17.45 1.57 0.034
CCNUOVAITOR ROME 258 14.73 10.60 0.96 0.792
CCCITTA’ROMA ROME 203 8.37 6.72 0.61 0.046
CCAURELIAHOSPITAL ROME 396 10.35 10.59 0.95 0.775
PRES.OSP.NORD LATINA 690 11.30 13.49 1.22 0.096
PRES.OSP.SUD FORMIA 500 11.80 13.16 1.19 0.204
CCCITTA’DIAPRILIA APRILIA 245 13.47 13.48 1.22 0.286
OSP.UMBERTOI FROSINONE 556 9.17 10.07 0.91 0.502
OSP.CIVILE ANAGNI 152 9.87 10.98 0.99 0.968
OSP.S.S.TRINITA’ SORA 220 10.91 11.44 1.03 0.885
OSP.G.DEBOSIS CASSINO 222 9.46 8.85 0.80 0.323
OSP.S.PERTINI ROME 763 11.40 11.99 1.08 0.491
OSP.BELCOLLE VITERBO 602 10.96 12.40 1.12 0.386
A.O.S.CAMILLO-FORLANINI ROME 1021 12.83 13.00 1.17 0.088
A.O.S.GIOVANNI ROME 679 7.81 7.52 0.68 0.007
A.O.S.FILIPPONERI ROME 770 10.91 11.58 1.04 0.708
POLICLINICOA.GEMELLI ROME 695 8.06 7.73 0.70 0.011
POLICLINICOUMBERTOI ROME 557 10.23 11.88 1.07 0.619
A.O.S.ANDREA ROME 615 4.88 6.16 0.55 0.001
A.O.POL.TORVERGATA ROME 505 5.94 9.67 0.87 0.451
RR:RelativeRisk
In Italy, initiatives aimed at assessing the outcomes of factorsor their effects do not vary betweenthe hospitals
hospital care have been undertaken at the national and beingcompared.
regionallevelsonlyinthelastdecade[13,17,40,41].Based After receiving comparative reports based on standar-
ontheseexperiences,wedevelopedtheRegionalOutcome dized performance measures, hospitals that began as
Evaluation Program, called P.Re.Val.E. [14]. The high low-level performers tended to improve faster than
numbersofpatientsinvestigated,theaccuracyintheselec- those that started at higher levels of performance [42].
tion of the cohorts and the study outcomes, the consoli- Sincestudieshaveshownlittlecorrelationbetweenmea-
dated statistical strategy, and the replication of similar sured quality of care and Standardized Mortality Ratios
findingsfordifferentclinicalconditionsareimportantele- (SMRs)[43,44],weuseddifferentoutcomeand/orprocess
ments of internal and external validity. For 2006-2009, measuresforeachstudiedconditiontobetteridentifythe
results were obtained using direct risk adjustment for hospitals/geographicareasthatneededhealthcarequality
comparative evaluation of outcomes for hospitals and improvement. This study used data from several health
areas of residence. Hospital league tables obtained by careinformationsystems,andmostoftheindicatorswere
indirectstandardizationproceduresshouldnotbeusedfor basedontheconceptoffirsthospitalaccess,correspond-
hospital-to-hospitalcomparisons[20].Thistechniquecan ingtopatientadmissiontoanacuteinpatientfacilityorto
lead to biased conclusions unless the distribution of risk emergency department access. The indicators for which
Fuscoetal.BMCHealthServicesResearch2012,12:25 Page6of10
http://www.biomedcentral.com/1472-6963/12/25
Table 3Acute myocardial infarction: mortality within 30 daysofhospital admission, by area ofresidence
Area N Cruderate×100 Adjustedrate×100 AdjustedRR p
RomeDistrictI 385 10.39 9.31 0.84 0.297
RomeDistrictII 312 10.90 8.80 0.79 0.202
RomeDistrictIII 129 15.50 12.39 1.12 0.641
RomeDistrictIV 567 9.70 9.04 0.82 0.152
RomeDistrictV 709 10.86 11.13 1.00 0.982
RomeDistrictVI 515 11.65 11.06 1.00 0.981
RomeDistrictVII 298 9.40 9.86 0.89 0.546
RomeDistrictVIII 565 9.03 10.96 0.99 0.932
RomeDistrictIX 351 12.25 10.06 0.91 0.548
RomeDistrictX 454 9.69 10.44 0.94 0.696
RomeDistrictXI 415 14.70 12.49 1.13 0.387
RomeDistrictXII 427 11.94 11.26 1.01 0.922
RomeDistrictXIII 700 11.43 10.92 0.98 0.891
Fiumicino 221 9.50 9.90 0.89 0.620
RomeDistrictXV 445 11.46 10.31 0.93 0.623
RomeDistrictXVI 424 12.97 11.52 1.04 0.794
RomeDistrictXVII 268 13.43 10.19 0.92 0.636
RomeDistrictXVIII 393 12.72 11.46 1.03 0.827
RomeDistrictXIX 594 11.62 10.05 0.91 0.444
RomeDistrictXX 414 11.84 12.84 1.16 0.329
ASLRM/F 894 10.07 10.04 0.90 0.368
ASLRM/G 1278 10.95 11.78 1.06 0.500
ASLRM/H 1693 10.16 10.92 0.98 0.846
Viterbo 182 9.34 8.77 0.79 0.347
ViterboProvince 741 12.42 13.08 1.18 0.132
Rieti 181 9.39 9.37 0.84 0.504
RietiProvince 329 13.37 13.16 1.19 0.281
Latina 307 11.73 12.45 1.12 0.511
LatinaProvince 1223 11.37 12.73 1.15 0.124
Frosinone 180 11.67 13.71 1.24 0.345
FrosinoneProvince 1088 9.93 10.62 0.96 0.662
RR:RelativeRisk
timetodeathortosurgerywascalculatedfromfirsthospi- The P.Re.Val.E. results for 2008-2009 [14] provided an
tal access provide a measure of the appropriateness and overview of the hospital care heterogeneity in the Lazio
efficacy of the health care process that begins when a region. The results do not constitute a “league table” of
patientarrivestoagivenfacility.UseoftheMIS andEIS performance, but instead reveal numerous instances of
databases,inaddition to theHIS database, allowedmore high-quality care as well as problem areas that merit
accurateidentificationof30-daymortality. further analysis and internal and external auditing as
To monitor the time trends of the different outcomes, part of an increasingly well-developed program of clini-
we used VLAD charts, a type of quality control chart cal governance.
that is a good tool for measuring the variability of an P.Re.Val.E. is an outcome research program conceived
event adjusting for patient risk [34,35,45,46]. VLAD mainly as a tool for promoting discussion among health-
charts provide an easy-to-understand and up-to-date care managers and professionals in the Lazio region.
view that allows early detection of runs of good or bad Given the complexities of accurately comparing provider
outcomes and thus can prompt timely intervention for outcomes, we published the methods used for develop-
critical situations. The VLAD charts also highlight small ing the program in detail, using others’ suggestions for
variations over time for observed events compared to the public reporting of comparative health outcome eva-
expected events; this information is often obscured by luations [47] so that the face validity of the results could
the corresponding synthetic indicator. be evaluated. We also used various tools to present the
Fuscoetal.BMCHealthServicesResearch2012,12:25 Page7of10
http://www.biomedcentral.com/1472-6963/12/25
Figure1VLADchart.Acutemyocardialinfarction:mortalitywithin30daysofhospitaladmission.HospitalPresidioOspedalieroNord,Latina.
results in order to make the results accessible; in parti- aimed at creating an incentive to improve results. In
cular, we used bar graphs to clearly display adjusted fact, studies describing the effect of public reporting on
estimates, as well as tables that included the number of consumers’ choices, effectiveness, patient safety and
admissions, the crude and adjusted estimates, and the patient-centeredness have shown that the public release
statistical significance of the results. of performance data mainly stimulates change at hospi-
Public disclosure of the 2006-2009 results to clini- tal level [48]. There have been some negative reactions,
cians, health care managers, and policy makers was but in most cases, the results have stimulated increased
Figure2VLADchart.Acutemyocardialinfarction:mortalitywithin30daysofhospitaladmission.HospitalS.FilippoNeri,Rome.
Fuscoetal.BMCHealthServicesResearch2012,12:25 Page8of10
http://www.biomedcentral.com/1472-6963/12/25
review among health care organizations and profes- the P.Re.Val.E. update for 3 conditions, namely AMI,
sionals. Clinicians and program developers have met aortocoronarybypass,andhipfracture,sincesomeclini-
several times to discuss the methodology as well as cal information (e.g., systolic blood pressure, ejection
negative findings, such as poor performance or poor fraction,creatinine)havebeenrecentlyaddedtotheHIS
coding accuracy. Physicians have made suggestions [60]. Moreover, in these 3 conditions, intrinsic illnesses
about more accurate selection criteria for some indica- at the time of patients’ admissions can be distinguished
tors. As a consequence of the extremely low proportion fromcomplicationssince “present on admission”(POA)
of interventions for hip fracture in the elderly within 48 flags have been added to discharge diagnoses [61]. The
hours in most facilities, the regional authority decided problem of under-recording is also partially solvable by
that hospitals with performance results below a given using prior patient hospitalization records to identify
standard would be penalized economically by a reduc- comorbidities independent of patient severity at the
tion in fees corresponding to specific Diagnosis Related currentadmission[61] as wellas emergencydepartment
Groups (DRGs) [49]. visitstocollectadditional informationaboutpatientrisk
P.Re.Val.E. is a program in progress, and it will be factors. However, the coding accuracy may differ widely
updatedandfurtherdevelopedbythedefinitionandcal- among the facilities [62], and this could lead to biased
culationofadditionalindicators,e.g.,thoseaimedateval- comparisons. Even though the possibility of gaming of
uating health care quality for oncology patients, and by thedatainresponsetothe performance evaluation can-
the use of regional drug dispensing registries to more not be excluded, previous studies did not find evidence
accurately identify patient comorbidities. The impact of ofgaming[63].Somestudieshavereportedthatchanges
health care performance information disclosure to the in data accuracy may partially explain quality improve-
general public should be evaluated. Evidence suggests ment[64].However,wedidnotfindrelevantchangesin
thatthisinformationhasonlyalimitedimpactonconsu- recordingofco-morbiditiesinourstudypopulationover
mer decision-making [39] since people have limited the years (data not shown). In agreement with previous
access to data on health care providers [50]; however, reports,theprevalenceofcertainco-morbiditiesandrisk
studiessuggestthatpeopleareinterestedincomparative factors was relatively low in our study population, indi-
information[51,52]. cating underreporting of co-morbidities and detailed
Theseanalyseshaveexplicitlimitations,especiallywith clinical information in the administrative database [65].
regardtothemarkedvariabilityinthecodingaccuracyof However, underreporting was non-differential in the
currenthealthcareinformationsystems.Thisissueiscri- yearsincludedinouranalysis.Moreover,a previousIta-
tical for ensuring accurate risk adjustment, and, corre- lian study [65] assessing clinical performance in cardiac
spondingly, reliable comparative quality ratings [53]. In surgery demonstrated that the use of an administrative
the past, administrative databases have too frequently database provided similar league tables as a more com-
beenusedexclusivelyastoolstoclaimfinancialreimbur- plexspecializeddatabase.
sement for services provided without concern for their Finally,sincehealthcareservicescanonlybeevaluated
roles as epidemiologic sources and as essential instru- byempiricalmeasurements,inevitablytherewillbeerrors
mentsforclinicalgovernance.Therearesomeimportant (systematicandrandom).Thisrepresentsaclearlimitation
advantages to routinely collectingadministrative data:it ofouranalysisaswellasothersofthistype.Weagreewith
is inexpensive to do this, and the data provide informa- Shahianetal.thathospitalmortalityestimatescouldvary,
tionaboutlargepopulations,donotdependonvoluntary sometimes widely, based on the different case-selection
participation by individual clinicians and providers, and criteriaandstatisticalmethods,leadingtodivergentinfer-
can be used to predict risk of death with discrimination ences about relative hospital performance. Despite these
comparable with that obtained from clinical databases concerns,somefindingscouldbeusefultopotentialusers
[54].However,theuseofroutinelycollectedadministra- or to facilities [66]. P.Re.Val.E. openly declares the data
tive data in comparative outcome evaluations has been sources and methods, allowing external review of biases
criticizedforthefollowingreasons:thereisanabsenceof and distortions implicit to the evaluation process. The
clinical information needed to adequately adjust for nextP.Re.Val.E.analysis,expectedinNovember2011,will
patients’conditions[55,56];thereisaninabilitytodistin- include improvements in methods and procedures; of
guish between a disease present at admission (a comor- course,we cannotsaythatotherbiases willnotbeintro-
bidity, i.e. a true patient risk factor) and one that duced,simplythatthey willbedifferent.It isourconvic-
occurred during the hospital stay (i.e. a complication) tion, however, that P.Re.Val.E. is an important operative
[56,57] and some chronic comorbidities, such as hyper- toolthatshouldbeusedtopromoteclinicalandorganiza-
tension and diabetes, are known to be currently under- tionalmonitoringofhealthcareproviders,tosupportpoli-
reported at admission, mainly in more severely affected ticaldecision-making processes,andto stimulate asense
patients[58,59].Thefirstproblemcouldbeovercomein of healthy, productive competition aimed at improving
Fuscoetal.BMCHealthServicesResearch2012,12:25 Page9of10
http://www.biomedcentral.com/1472-6963/12/25
healthcareefficacyandequity.Wehopethatthisprogram InfarctionQualityIndicatorPanel.Indicatorsofqualityofcarefor
will encourage a value often neglected within the Italian patientswithacutemyocardialinfarction.CMAJ2008,179(9):909-915.
10. HealthGrades.TheSeventhAnnualHealthGradesHospitalQualityand
NHS:accountability.
ClinicalExcellenceStudy. Lakewood,CO:HealthGrades2009[http://www.
healthgrades.com/media/DMS/pdf/
HospitalQualityClinicalExcellenceStudy2009.pdf].
Additional material
11. EUPHORIC-EUropeanPublicHealthOutcomeResearchandIndicators
Collectionproject.[http://www.euphoric-project.eu].
Additionalfile1:Appendix:operativeprotocol.30-DayMortalityRate 12. MattkeS,KelleyE,SchererP,HurstJ,LapetraMLG,theHCQIExpertGroup
afterhospitaladmissionforAcuteMyocardialInfarction(AMI). Members:HealthCareQualityIndicatorsProjectInitialIndicatorsReport.
OECDHealthWorkingPapers2006,22.
13. MinistryofHealth,Italy.Mattoni-outcomeproject.[http://www.mattoni.
salute.gov.it/].
14. LazioRegionalHealthService(RHS)OutcomeEvaluationProgram2006-
Acknowledgements
2009.,http://www.epidemiologia.lazio.it/vislazio/vis_index.phpandhttp://
WewouldliketothankFedericaAsta,PaolaColais,MirkoDiMartino,and
www.epidemiologia.lazio.it/prevale10/index.php.
MartinaVenturafortheirsupport.
15. LazioRegion.Resolutionno.301. OfficialBulletinoftheLazioRegion2008,
25(77supp.).
Authordetails
16. NationalInstituteofStatistics(ISTAT),Italy.[http://demo.istat.it/pop2007/
1DepartmentofEpidemiology,RegionalHealthService-LazioRegion,via
index_e.html].
SantaCostanza53,Rome,00198,Italy.2NationalAgencyofRegionalHealth
17. AgabitiN,AnconaC,ForastiereF,ArcaM,PerucciCA:Evaluatingoutcomes
Services,viaPuglie23,Rome,00187,Italy.
ofhospitalcarefollowingcoronaryarterybypasssurgeryinRome,Italy.
EurJCardiothoracSurg2003,23(4):599-606,discussion607-608.
Authors’contributions
18. FantiniMP,StivanelloE,FrammartinoB,BaroneAP,FuscoD,DallolioL,
DFconceivedtheidea,participatedinthedesign,analysis,interpretationof
Cacciari,PerucciCA:Riskadjustmentforinter-hospitalcomparisonof
dataandhavebeeninvolvedindraftingthemanuscript.APBparticipatedin
primarycesareansectionrates:need,validityandparsimony.BMCHealth
thedesign,interpretationofdataandcontributedtothemanuscript
ServRes2006,6(1):100.
preparation.
19. CaponA,DiLalloD,PerucciCA,PanepucciaL:Case-mixadjustedodds
CSconductedthestatisticalanalysis,providedtechnicalsupportand
ratiosasanalternativewaytocomparehospitalperformances.EurJ
contributedtothemanuscriptpreparation.MDmadesubstantial
Epidemiol2005,20(6):497-500.
contributionstoperformthestatisticalanalysis.MSconductedthestatistical
20. ClaytonD,HillsM:StatisticalModelsinEpidemiology.NewYork:Oxford
analysisandprovidedtechnicalsupport.ALconductedthestatisticalanalysis
UniversityPress;1993.
andprovidedtechnicalsupport.MDparticipatedinthedesignand
21. AgencyforHealthcareResearchandQuality(AHRQ).AHRQquality
interpretationofdata.CAPparticipatedinthedesign,coordinationand
indicators.Guidetopreventionqualityindicators. Rockville,MD:Agency
interpretationofdata.Allauthorsreadandapprovedthefinalmanuscript.
forHealthcareResearchandQuality.Version3.12007[http://www.
qualityindicators.ahrq.gov/].
Competinginterests
22. CaliforniaHospitalOutcomesProject:HeartAttackOutcomes1996-1998.
Theauthorsdeclarethattheyhavenocompetinginterests.
Volume2:TechnicalGuide.OSHPD2002.
23. CCORT/CCSAcuteMyocardialInfarctionCanadianQualityIndicator
Received:12January2011 Accepted:27January2012
Panel.[http://www.ccort.ca/].
Published:27January2012
24. CommissionforHealthImprovement.Performanceindicators&ratings
for2002/2003IndicatorlistingsforAcutetrusts.[http://www.chi.nhs.uk/
References Ratings/Trust/Indicator/indicators.asp?trustType=1].
1. RelmanAS:Assessmentandaccountability:thethirdrevolutionin 25. ChampneyKP,FrederickPD,BuenoH,ParasharS,FoodyJ,BaireyMerzCN,
medicalcare.NEnglJMed1988,319:1220-1222. CantoJG,LichtmanJH,VaccarinoV:TheJointContributionofSex,Age
2. MarshallMN,ShekellePG,LeathermanS,BrookRH:Thepublicreleaseof andTypeofMyocardialInfarctiononHospitalMortalityFollowingAcute
performancedata:whatdoweexpecttogain?areviewofthe MyocardialInfarction.Heart2009,95(11):895-899.
evidence.JAMA2000,283:1866-1874. 26. NewYorkStateDepartmentofHealth.Percutaneouscoronary
3. GibberdR,HancockS,HowleyP,RichardsK:Usingindicatorstoquantify interventions(PCI)inNewYorkState2004-2006. Albany,NewYork:New
thepotentialtoimprovethequalityofhealthcare.IntJQualHealthCare YorkStateDepartmentofHealth;2009[http://www.health.state.ny.us/
2004,16:i37-i43. statistics/diseases/cardiovascular/].
4. BerwickDM,JamesB,CoyeMJ:Connectionsbetweenquality 27. UgoliniC,NobilioL:Riskadjustmentforcoronaryarterybypassgraft
measurementandimprovement.MedCare2003,41:I30-I38. surgery:anadministrativeapproachversusEuroSCORE.IntJQualHealth
5. JointCommissiononAccreditationofHealthcareOrganizations(JCAHO): Care2004,16(2):157-164.
SpecificationsManualforNationalHospitalInpatientQualityMeasures, 28. GudmundssonG,GislasonT,LindbergE,HallinR,UlrikCS,BrondumE,
version4.0c.JCAHO2011[http://www.jointcommission.org/ NieminenMM,AineT,BakkeP,JansonC:MortalityinCOPDpatients
performance_measurement.aspx]. dischargedfromhospital:theroleoftreatmentandco-morbidity.Respir
6. LeathermanS,SutherlandK:ThequestforqualityintheNHS:refiningthe Res2006,7:109.
NHSreforms.London.TheNuffieldTrust2008,Availableat:https://www. 29. LöfvendahlS,EckerlundIr,HansagiH,MalmqvistB,ReschS,HanningM:
evidence.nhs.uk. Waitingfororthopaedicsurgery:factorsassociatedwithwaitingtimes
7. AgencyforHealthcareResearchandQuality(AHRQ):AHRQquality andpatients’opinion.IntJQualHealthCare2005,17(2):133-140.
indicators.Guidetoinpatientqualityindicators:qualityofcarein 30. FrenchDD,BassE,BradhamDD,CampbellRR,RubensteinLZ:
hospitals-volume,mortality,andutilization,version3.1.Rockville(MD): Rehospitalizationafterhipfracture:predictorsandprognosisfroma
AgencyforHealthcareResearchandQuality2007[http://www. nationalveteransstudy.JAmGeriatrSoc2008,56(4):705-710.
qualityindicators.ahrq.gov/]. 31. IezzoniLI:RiskAdjustmentformeasuringhealthcareoutcomes.Health
8. NHS-NationalCentreforHealthOutcomesDevelopment.Clinicaland AdministrationPress;,32003.
HealthOutcomesKnowledgeBaseCareQualityCommission.[http:// 32. ArcàM,FuscoD,BaroneAP,PerucciCA:Riskadjustmentandoutcome
www.nchod.nhs.uk/]. research.PartI.JCardiovascMed(Hagerstown)2006,7(9):682-690.
9. TuJV,KhalidL,DonovanLR,KoDT:CanadianCardiovascularOutcomes 33. ArcàM,FuscoD,BaroneAP,ePerucciCA:Introductiontoriskadjustment
ResearchTeam/CanadianCardiovascularSocietyAcuteMyocardial methodsincomparativeevaluationofoutcomes.EpidemiolePrev2006,
30(4-5suppl1):1-48,initalian.
Fuscoetal.BMCHealthServicesResearch2012,12:25 Page10of10
http://www.biomedcentral.com/1472-6963/12/25
34. LovegroveJ,ValenciaO,TreasureT,Sherlaw-JohnsonC,GallivanS: ratesafteradmissionforacutemyocardialinfarction.MedCare2007,
Monitoringtheresultsofcardiacsurgerybyvariablelife-adjusted 45:1180-1185.
display.Lancet1997,350(9085):1128-1130. 56. PineM,JordanHS,FryDE,HoaglinDC,JonesB,MeimbanR,WarnerD,
35. FoltranF,BaldiI,BertoliniG,MerlettiF,GregoriD:Monitoringthe GonzalesJ:Enhancementofclaimsdatatoimproveriskadjustmentof
performanceofintensivecareunitsusingthevariablelife-adjusted hospitalmortality.JAMA2007,297(1):71-76.
display:asimulationstudytoexploreitsapplicabilityandefficiency.J 57. GlanceLG,OslerTM,MukamelDB,DickAW:Impactofthepresent-on-
EvalClinPract2009,15(3):506-513. admissionindicatoronhospitalqualitymeasurement:experiencewith
36. SASInstitute,Inc.SAS/STATsoftware,version8. Cary,NC:SASInstitute, theAgencyforHealthcareResearchandQuality(AHRQ)Inpatient
Inc;1999. QualityIndicators.MedCare2008,46(2):112-119.
37. BradleyEH,HolmboeES,MatteraJA,RoumanisSA,RadfordMJ, 58. QuanH,ParsonsGA,GhaliWA:Validityofinformationoncomorbidity
KrumholzHM:Datafeedbackeffortsinqualityimprovement:lessons derivedfromICD-9-CMadministrativedata.MedCare2002,40(8):675-685.
learnedfromUShospitals.QualSafHealthCare2004,13:26-31. 59. ElixhauserA,SteinerC,HarrisDR,CoffeyRM:Comorbiditymeasuresfor
38. MarshallMN,ShekellePG,DaviesHTO,SmithPC:Publicreportingon usewithadministrativedata.MedCare1998,36(1):8-27.
qualityintheUnitedStatesandtheUnitedKingdom.HealthAff 60. Modificadelcontenutoinformativodellarilevazionedeidatisulle
(Millwood)2003,22(3):134-148. dimissioniospedaliereperpazientiacuti. LazioRegion2007,Resolution
39. MarshallMN,ShekellePG,LeathermanS,BrookRH:ThePublicReleaseof D4118http://www.asplazio.it/.
PerformanceData.WhatDoWeExpecttoGain?AReviewofthe 61. LeeDS,DonovanL,AustinPC,GongY,LiuPP,RouleauJL,TuJV:
Evidence.JAMA2000,283(14):1866-1874. Comparisonofcodingofheartfailureandcomorbiditiesin
40. UgoliniC,NobilioL:Riskadjustmentforcoronaryarterybypassgraft administrativeandclinicaldataforuseinoutcomeresearch.MedCare
surgery:anadministrativeapproachversusEuroSCORE.IntJQualHealth 2005,43(2):182-188.
Care2004,16:157-164. 62. RangachariP:CodingforQualityMeasurement:theRelationship
41. SeccarecciaF,PerucciCA,D’ErrigoP,ArcàM,FuscoD,RosatoS,GrecoD, betweenHospitalStructuralCharacteristicsandCodingAccuracyfrom
ResearchGroupoftheItalianCABGOutcomeStudy:TheItalianCABG thePerspectiveofQualityMeasurement.PerspectHealthInfManag2007,
OutcomeStudy:short-termoutcomesinpatientswithcoronaryartery 4:3.
bypassgraftsurgery.EurJCardiothoracSurg2006,29:56-62. 63. SerumagaB,Ross-DegnanD,AveryAJ,etal:Effectofpayforperformance
42. WilliamsSC,SchmaltzSP,MortonDJ,KossRG,LoebJM:QualityofCarein onthemanagementandoutcomesofhypertensionintheUnited
U.S.HospitalsasReflectedbyStandardizedMeasures,2002-2004.NEngl Kingdom:interruptedtimeseriesstudy.BMJ2011,342:d108.
JMed2005,353:255-264. 64. WernerRM,KonetzkaRT,StuartEA,etal:Impactofpublicreportingon
43. PitchesDW,MohammedMA,LilfordRJ:Whatistheempiricalevidence qualityofpostacutecare.HealthServRes2009,44:1169-87.
thathospitalswithhigher-riskadjustedmortalityratesprovidepoorer 65. FortunaD,VizioliM,ContiniA,etal:Assessingclinicalperformancein
qualitycare?Asystematicreviewoftheliterature.BMCHealthServRes cardiacsurgery.Doesaspecialisedclinicaldatabasemakeadifference?
2007,20:91. InteractCardiovascThoracSurg2006,5:123-127.
44. LilfordR,PronovostP:Usinghospitalmortalityratestojudgehospital 66. ShahianDM,WolfRE,IezzoniLI,KirleL,NormandSLT:Variabilityinthe
performance:abadideathatjustwon’tgoaway.BMJ2010,340:c2016. MeasurementofHospital-wideMortalityRates.NEnglJMed2010,
45. WinkelP,ZhangNF:Risk-AdjustedControlCharts.StatisticalDevelopment 363:2530-9.
ofQualityinMedicineJohnWiley&Sons,Ltd;2007,149-163.
46. CooryM,DuckettS,Sketcher-BakerK:Usingcontrolchartstomonitor Pre-publicationhistory
qualityofhospitalcarewithadministrativedata.IntJQualHealthCare Thepre-publicationhistoryforthispapercanbeaccessedhere:
2008,20(1):31-39. http://www.biomedcentral.com/1472-6963/12/25/prepub
47. KrumholzHM,BrindisRG,BrushJE,CohenDJ,EpsteinAJ,FurieK,
HowardG,PetersonED,RathoreSS,SmithSCJr,SpertusJA,WangYand doi:10.1186/1472-6963-12-25
NormandS-LT:StandardsforStatisticalModelsUsedforPublicReporting Citethisarticleas:Fuscoetal.:P.Re.Val.E.:outcomeresearchprogram
ofHealthOutcomes.AnAmericanHeartAssociationScientificStatement fortheevaluationofhealthcarequalityinLazio,Italy.BMCHealth
FromtheQualityofCareandOutcomesResearchInterdisciplinary ServicesResearch201212:25.
WritingGroup.Circulation2006,113:456-462.
48. ShekellePG,LimYW,MattkeS,DambergC:Doespublicreleaseof
performanceresultsimprovequalityofcare?Asystematicreview.
London.TheHealthFoundation2008[http://www.health.org.uk/
publications/does-public-release-of-performance-results-improve-quality-of-
care/#].
49. LazioRegion.Resolutionno.613. OfficialBulletinoftheLazioRegion2009,
34(163supp.).
50. MannionR,GoddardM:Publicdisclosureofcomparativeclinical
performancedata:lessonsfromtheScottishexperience.JEvalClinPract
2003,9:277-286.
51. SofaerS,CroftonC,GoldsteinE,HoyE,CrabbJ:Whatdoconsumerswant
toknowaboutthequalityofcareinhospitals?HealthServRes2005,
40:2018-2036.
52. TheKaiserFamilyFoundation,AgencyforHealthcareResearchandQuality, Submit your next manuscript to BioMed Central
HarvardSchoolofPublicHealth:NationalSurveyonConsumers’ and take full advantage of:
ExperienceswithPatientSafetyandQualityInformation.Summaryand
ChartPack2004[http://www.kff.org/kaiserpolls/pomr111704pkg.cfm].
• Convenient online submission
53. MohammedMA,DeeksJJ,GirlingA,RudgeG,CarmaltM,StevensAJ,
LilfordRJ:Evidenceofmethodologicalbiasinhospitalstandardised • Thorough peer review
mortalityratios:retrospectivedatabasestudyofEnglishhospitals.BMJ
• No space constraints or color figure charges
2009,338:b780.
54. AylinP,BottleA,MajeedA:Useofadministrativedataorclinical • Immediate publication on acceptance
databasesaspredictorsofriskofdeathinhospital:comparisonof • Inclusion in PubMed, CAS, Scopus and Google Scholar
models.BMJ2007,334(7602):1044.
• Research which is freely available for redistribution
55. JohnstonTC,CooryMD,ScottI,DuckettS:Shouldweaddclinical
variablestoadministrativedata?Thecaseofrisk-adjustedcasefatality
Submit your manuscript at
www.biomedcentral.com/submit