Using relative survival measures for cross-sectional and longitudinal benchmarks of countries, states, and districts: the BenchRelSurv- and BenchRelSurvPlot-macros.

0.37 MB·English

by  Jacke, Christian O

Jackeetal.BMCPublicHealth2013,13:34
http://www.biomedcentral.com/1471-2458/13/34
SOFTWARE Open Access
Using relative survival measures for cross-sectional
and longitudinal benchmarks of countries, states,
and districts: the BenchRelSurv- and
BenchRelSurvPlot-macros
Christian O Jacke1*, Iris Reinhard1 and Ute S Albert2
Abstract
Background: The objective ofscreeningprograms is to discoverlife threatening diseases in as many patients as
early as possibleand to increasethe chance ofsurvival. To be able to compare aspectsof health care quality,
methods are needed for benchmarking thatallow comparisons onvarious health care levels (regional, national, and
international).
Objectives: Applicationsandextensionsofalgorithmscanbeusedtolinktheinformationondiseasephaseswith
relativesurvivalratesandtoconsolidatethemincompositemeasures.TheapplicationofthedevelopedSAS-macroswill
giveresultsforbenchmarkingofhealthcarequality.Dataexamplesforbreastcancercarearegiven.
Methods:Areferencescale(expected,E)mustbedefinedatatimepointatwhichallbenchmarkobjects(observed,O)
aremeasured.AllindicesaredefinedasO/E,wherebytheextendedstandardizedscreening-index(eSSI),the
standardizedcase-mix-index(SCI),thework-up-index(SWI),andthetreatment-index(STI)addressdifferenthealthcare
aspects.Thecompositemeasurescalledoverall-performanceevaluation(OPE)andrelativeoverallperformanceindices
(ROPI)linktheindividualindicesdifferentlyforcross-sectionalorlongitudinalanalyses.
Results:Algorithmsallowatimepointandatimeintervalassociatedcomparisonofthebenchmarkobjectsinthe
indiceseSSI,SCI,SWI,STI,OPE,andROPI.Comparisonsbetweencountries,statesanddistrictsarepossible.Exemplarily
comparisonsbetweentwocountriesaremade.Thesuccessofearlydetectionandscreeningprogramsaswellasclinical
healthcarequalityforbreastcancercanbedemonstratedwhilethepopulation’sbackgroundmortalityisconcerned.
Conclusions:Ifexternalqualityassuranceprogramsandbenchmarkobjectsarebasedonpopulation-basedand
correspondingdemographicdata,informationofdiseasephaseandrelativesurvivalratescanbecombinedtoindices
whichofferapproachesforcomparativeanalysesbetweenbenchmarkobjects.Conclusionsonscreeningprogramsand
healthcarequalityarepossible.Themacroscanbetransferredtootherdiseasesifadisease-specificphasescaleof
prognosticvalue(e.g.stage)exists.
Keywords:Benchmark*,Prevention&control,Outcomeassessment,Relativesurvival,Registries,Breastcancer
*Correspondence:[email protected]
1CentralInstituteofMentalHealth,MedicalFacultyMannheim/University
Heidelberg,SquareJ5,68159Mannheim,Germany
Fulllistofauthorinformationisavailableattheendofthearticle
©2013Jackeetal.;licenseeBioMedCentralLtd.ThisisanOpenAccessarticledistributedunderthetermsoftheCreative
CommonsAttributionLicense(http://creativecommons.org/licenses/by/2.0),whichpermitsunrestricteduse,distribution,and
reproductioninanymedium,providedtheoriginalworkisproperlycited.
Jackeetal.BMCPublicHealth2013,13:34 Page2of9
http://www.biomedcentral.com/1471-2458/13/34
Background thebenchmark-algorithms(SAS-macros)willstandardize
In many diseases early detection and secondary preven- the screening-index, produce a positive association with
tion are of central importance [1-3]. Life-threatening theobservationtimeafterappropriatemathematicalcon-
malignant diseases are therefore divided into stages version, and will adequately assess a composite measure.
according to their disease phase. They are recorded Disease-specific fatality rates will be replaced by relative
using international nomenclatures (e.g. Union for Inter- survival rates as outcome-indicators. The macro results
national Cancer Control, American Joint Committee in will be demonstrated by exemplary applications. The
Cancer) [4,5]. Thus, screening programs are aimed at development of the indices and the main working hy-
discovering diseases in as many patients and as early as pothesis is built on the assumption that comprehensive
possible toincreasetheirchance ofsurvival. early detection programs are capable of detecting dis-
First approaches on how the number of patients per eases early, leading to disease stage shifts and facilitates
disease phase/stage can be linked to the corresponding clinical work-up which is associated with increasing
chance of survival are based on Beatty et al. [6,7]. They relative 5-year survival rates. Improvements of health
developed a series of benchmark-algorithms that address care quality should be illustrated transparently.
different aspects of health care. Beatty and colleagues
described a screening index based upon the sum of the Implementations
products of the stage number (0–4) and the number of Data
cases at that stage divided by the total number of cases. Ingeneral,populationbaseddatabaseswithdemographic
Using the stage number in the calculation was consid- information such as registers (e.g. cancer register) may
ered arbitrary and it was replaced by the national 5-year be used. Clinical registers, cohort studies, or health care
mortality for that stage termed the case-mix index for network databases are also suited if a high data quality
the institution or region. This was then standardized by andepidemiologicrelevanceisensured[9-11].Theappli-
comparison to the national case-mix index and termed cation example is based on the assumption that data of
the standardized case-mix index (SCI). They also the Surveillance Epidemiology and End Results (SEER)
describedastandardizedwork-upindex(SWI)toaddress [12] and the Norwegian cancer register [13] fulfill these
theissueof ‘upstaging’ofcasesanda standardized treat- requirements.
ment index (STI) to evaluate outcomeusing institutional
orregionalmortality (stageandoverall)compared to na- Variables
tional mortalities. The product of the SCI, SWI and STI Thevariablesforthe“best-performer“,thatmustbeused
defined the composite measure named overall perform- to compare all others, define the reference object. All
ance evaluation (OPE). They used these indices and eva- other comparative objects are designated as benchmark
luations to comparedifferent regions over the same time objects and must have identification numbers that can
interval (cross-sectional analysis perspective) and the definitelybedistinguishedfromoneanother.Inaddition,
same institution across different time intervals (longitu- information on the year of first definite diagnosis (inci-
dinalanalysisperspective). dence year) and the disease phase (e.g. for cancer stage
However, the screening-index was not standardized, 0-IV) is necessary. The absolute and relative distribution
reflected a negative association with the observation ofpersonsperdiseasephase(e.g.perstage)aswellasthe
period, and, thus, was not included in the OPE. In corresponding relative survival rates (e.g. 5 years) are
addition, the OPE had a logical weakness, because the needed. In addition, the total number of ill patients per
standardized case-mix index was present in the denom- time unit (e.g. year) and the relative overall survival are
inator as well as the numerator and cancelled itself out. important.Table1servesasanexamplewhat specifically
Finally,theapproach wasbased ondisease-specific cause theSAS-MacroBenchRelSurvexpectsasavariable-setin
of death statistics which required the modeling of com- theso-calledlong-formatofSAS.
peting risks according to the method of Gooley [8]. This
method focuses on the mortality of one disease only Relativesurvivalasoutcome
and, therefore reduces the case numbers. This is of spe- If actually observed and expected chances of survival
cial importance for small geographic units that already probabilities are related to each other, relative survival
have small case numbers as it is. Thisoutcome-indicator rates are obtained [14]. The former originate from em-
alsodoesnotincludebackground-mortality. pirical data (e.g. registers). But in contrast, the latter can
be calculated from so-called “prospective probability of
Objectives death”from period or cohort life tables stratified accord-
Thiscontributionintendstoovercometheexistingshort- ing to age, gender, calendar year and occasionally also
comings of the proposed algorithms and to extend the ethnicity [15,16]. These life tables are available from the
indices (see methods). Furthermore, the automation of Federal Agency for Statistics or the Human Mortality
Jackeetal.BMCPublicHealth2013,13:34 Page3of9
http://www.biomedcentral.com/1471-2458/13/34
Table1ConfigurationofDataandVariableSets
Group Year Identification Stage NumberperStage NumberTotal RelativeSurvival
RefGroup RefYear RefID RefStage RefNpStage RefN RefRelSurv
SEER17 1999-2003 1 0 167 191771 93.6
SEER17 1999-2003 1 1 83081 191771 100.00
SEER17 1999-2003 1 2 72195 191771 90.2
SEER17 1999-2003 1 3 12617 191771 61.3
SEER17 1999-2003 1 4 8449 191771 22.5
SEER17 1999-2003 1 5 15262 191771 77.3
SEER17 1999-2003 1 6 191771 191771 89.1
Legend:(PrefixRef)Definitionforreferenceobjectsubstitutablebyprefix'Bench'indicatingbenchmarkobject,(Stage)Stageofdiseasewhere5='notknown'and
6='total'overallrespectively.
Source:NewmalignantbreastcancercasesfromSurveillance,Epidemiology,andEndResults(SEER)Program(www.seer.cancer.gov)SEER*Stat.
Database:Incidence-SEER17-Nov2010Sub(1973–2008varying).
Database (HMD). Different methods for the handling of summed up and divided by the total number of ill
censored cases [17,18] or various observation periods patients. The index is standardized by comparing bench-
[19-21] are possible. Non-parametric methods to derive marktoreferenceobjectsandisdefinedas:
relative survival rates are sufficient in this case and can
be easily estimated using freely accessible software- SCI¼(cid:2)O(cid:2)=XE(cid:2) (cid:3) (cid:3) .(cid:2)(cid:2)X (cid:3) (cid:3)
¼ N i(cid:3)RSR i =N O N i(cid:3)RSR i =N E(cid:2):
solutions(e.g.[22]).
Standardizedindices Standardizedwork-Upindex(SWI)
All indices have the principle of construction in common. ThecentralideaoftheSWIistorelatetherelativesurvival
Time and factually-fixed reference objects define the com- rate per stage (RSR_i) of a benchmark object (Obs) with
parative scale (here USA, 2003). This, so called reference the RSR_i of the reference object (E*). The resulting pro-
object is the best possible expected result (expected, Exp*). portionsarethensummarized.Finally,togetanideaofthe
Allotherbenchmarkobjects(observed,Obs)mustbecom- average relative survival rate across the stages, the sum is
pared to this reference scale. Hence, all indices are defined divided by the number of represented disease stages i. The
as: index = Obs/Exp*. From a cross-sectional analysis per- same is true for the reference object. This index is defined
spective, the benchmark objects (Obs) belong to the same as:
timeintervalasthereferenceobject(Exp*).Fromalongitu- (cid:2)X(cid:2) (cid:3) .(cid:2) (cid:3). (cid:2)(cid:3)
dinal analysis perspective, however, the benchmark objects SWI¼O=E(cid:2)¼ RSR i RSR iÞE(cid:2) N i :
O
can originate from different time intervals (Obs_t), while
thereferenceobjectisfixedintime.Tosimplifythematter, Standardizedtreatmentindex(STI)
thetimeindextisomittedinthefollowing. The central idea of the STI is to set the overall relative
survival rate (RSR) of benchmark objects (O) and refer-
Extendedstandardizedscreeningindex(eSSI) ence objects (E*) in relation to each other. However,
The central idea of the eSSI is focused on the relative pro- since benchmark objects and reference objects can differ
portionofillpersonsperdiseasestage(N_i/N)weightedby in their stage distribution, the SCI is needed as an indi-
thestagenumberitself.Theproductsarethensummedup cator ofriskadjustment.Theindex isdefined as:
forthebenchmarkobject.Thesumisthenputintorelation
with the sum of the reference object which characterizes
thestandardization-process.Theindexisdefinedas: STI¼O=E(cid:2)¼ðRSR Obs=RSR Exp(cid:2)Þ(cid:3)1=SCI obs
eSSI¼OhX=E(cid:2) i .h(cid:2) i Compositemeasures
¼ ðN i=NÞ(cid:3)i N i=NÞ(cid:3)i E(cid:2): According to Beatty et al. [6,7] SCI, SWI, and STI may
O be summarized to an overall performance evaluation
(OPE): OPE= SCI × SWI × STI. As an alternative the
Standardizedcase-Mixindex(SCI) relative overall performance index (ROPI) is suggested.
The central idea of the SCI is to multiply the absolute TheROPI isdefinedas:
number of ill persons per disease stage (N_i) with the
relative survival rate (RSR_i). The products are then ROPI¼1=eSSI(cid:3)SWI(cid:3)STI:
Jackeetal.BMCPublicHealth2013,13:34 Page4of9
http://www.biomedcentral.com/1471-2458/13/34
Exampleofuse If STI>1 is true, then the stage adjusted overall sur-
Thedataofnewmalignantbreastcancerpatientsfromthe vival rates in the benchmark object will be higher than
Surveillance Epidemiology and End Results (SEER17- in the reference object. If STI<1 is true, then the stage
Nov2010) [12] and the Norwegian cancer register [13] adjusted overall survival rates in the benchmark object
were used. The national SEER17-values from 1999–2003 are lower than in the reference object. The latter result
served as reference objects in the cross-sectional analyses. might be interpreted as a less effective overall treatment
Thus, these analyses were restricted to the period from than inthereferenceobject.
1999–2003and the seventeen SEER-registers which define If OPE or ROPI>1 is valid, then patients in the bench-
the benchmark objects. In the longitudinal analyses, the mark object will have earlier treatment with higher rela-
national SEER17-values from the last available year 2003 tive survival rates on average than in the reference
servedasreferenceobject.ThenationalSEER17datafrom object. If OPE or ROPI<1 is true, then patients in the
1990–2003 as well as the Norwegian data from the inter- benchmark object will have later treatment with lower
vals 1969–73, 1974–78, 1979–1983, 1984–88, 1989–93, relative survival rates on average than in the reference
1994–98, and 1999–03 [13] served as benchmark objects. object. Table 2 shows the cross-sectional results. Exam-
The relative 5-year survival rate was calculated according ples of longitudinal results for the eSSI and ROPI are
totheEdererII-method.Theexamplesareavailablewithin depicted in the Figures 1 and 2. The interpretation fol-
theadditionalfiles(Additionalfile1:Example1,Additional lows thegeneral instructions.
file2:Example2)andcanbedownloadedfromtheproject-
homepage(http://sourceforge.net/projects/benchrelsurv/). Discussion
The existing, updated and extended standardized indices
form an additional tool for the evaluation of health care
Results services and quality. Reference objects can be defined
Inthe cross-sectionalandlongitudinalapplicationexam- and compared to benchmark objects such as countries,
ples, the reference object was also included as bench- states, and districts. The indices offer a cross-sectional
mark object. This leads to a special case because and longitudinal perspective on benchmark objects. Es-
benchmark object and reference object are equal. There- pecially the latter offer the opportunity to demonstrate
fore, the corresponding indices are of value one in the relationship between disease stage and chance of
cross-sectional analysis. In longitudinal analyses, it leads survivalduringthe courseoftime.
to index values of one in the chosen reference year (here
2003). This logic allows for all the other benchmark Comparisonwithotherbenchmarking-algorithms
objects that a health service quality gap – or advance – The performed cross sectional analysis detects the already
is identified by index values smaller or larger than one. observed high variability between urban-metropolitan
If, for example, eSSI>1 is valid, more patients will be areas and rural regions which have led to controversial
treated at a later time point in the benchmark object discussions[23,24].Thelongitudinalanalyses,however,
than in the reference object. If eSSI<1 is valid, more may show the growing influence of comprehensive
people will be treated at an earlier time point than in early detection and screening-programs and -methods,
the reference object. The latter result might be inter- guidelines as well as increasing utilization and partici-
preted as a more effective early detection and screening pation rates that may lead to more favorable surrogate
programthaninthereferenceobject. parameters such as absolute or relative stage distribu-
In analogy, for SCI>1, the survival conditions in the tion. In this respect the approach is very similar to
benchmark objects will adapt to the normal population’s purely descriptive benchmark projects [2,3,25-28],
faster than in the reference object. If SCI<1 is measured, which also have to interpret obtained results within
the survival conditions in the benchmark object will country- and time-specific conditions. From this per-
adapt more slowly to the conditions of the population spective both approaches (presented benchmark,
than in the reference object. The latter result might be descriptives) aresomewhat complementarybecausethe
interpreted as a less effective early detection and screen- latter may provide insights in explicitly measured and
ingprogramthaninthereference object. process-related quality indicators which are based on
If SWI>1 is true, then the resumed average survival medical decisions. In exchange, the formulated bench-
rates across the stages will be higher in the benchmark mark algorithms reject the use of criteria-based
object than in the reference object. If SWI<1 is true, approaches to estimate theoretically expected cases
then the average survival rates across the stages will be [1,5,29] in the indices’ denominator. Therefore, the
lowerinthebenchmarkobjectsthaninthereference ob- here presented algorithms cannot inform about insuffi-
ject. The latter result might be interpreted as a less ef- cient respectively inappropriate health care. Further-
fective clinicalwork-upthaninthereference object. more, the proposed approach cannot revise short-term
Jackeetal.BMCPublicHealth2013,13:34 Page5of9
http://www.biomedcentral.com/1471-2458/13/34
Table2Cross-SectionalBenchmarkingofSEER17-Registers(1999–2003)
eSSI SCI SWI STI OPE ROPI
ReferenceObject(1999–2003)
SEER17 1.000 1.000 1.000 1.000 1.000 1.000
BenchmarkObjects
Iowa 0.951 0.993 1.032 1.023 1.048 1.110
NewMexico 0.973 0.959 1.023 1.037 1.017 1.090
Seattle(PugetSound) 0.960 1.035 1.033 1.004 1.074 1.080
NewJersey 0.991 0.938 0.997 1.052 0.983 1.059
Utah 0.991 0.987 1.026 1.010 1.022 1.045
SanFrancisco-Oakland(SF) 1.002 1.036 1.039 0.987 1.062 1.023
SanJose-Monterey(SJM) 0.983 1.049 1.009 0.991 1.049 1.018
Conneticut 0.973 1.020 0.985 1.000 1.005 1.013
Californiaexcl.SF/SJM/LA 0.994 1.018 1.006 0.990 1.014 1.002
AlaskaNatives 0.981 0.932 0.956 0.980 0.874 0.955
Atlanta 1.024 1.001 0.982 0.982 0.965 0.941
LosAngeles(LA) 1.051 1.018 1.008 0.978 1.003 0.938
Kentucky 0.965 0.927 0.862 1.045 0.834 0.933
Detroit(Metropolitan) 1.029 0.981 0.935 0.995 0.913 0.905
Hawaii 0.975 1.078 0.926 0.948 0.947 0.900
Lousiana 1.074 0.960 0.952 0.975 0.891 0.865
RuralGeorgia 0.947 0.884 0.709 1.076 0.674 0.805
Legend:(eSSI)ExtendedStandardizedScreeningIndex,(SCI)StandardizedCase-MixIndex,(SWI)StandardizedWork-UpIndex,(STI)StandardizedTreatmentIndex,
(OPE)OverallPerformanceEvaluation,(ROPI)RelativeOverallPerformanceIndex.
Source:NewmalignantbreastcancercasesfromSurveillance,Epidemiology,andEndResults(SEER)Program(www.seer.cancer.gov)SEER*Stat.
Database:Incidence-SEER17-Nov2010Sub(1973–2008varying).
outcomessuchasmorbidityor30-dayin-housemortal- differentiation purposes. Finally, it should be stressed
ity [4,30], because the chosen outcome parameters of that the developed approach extends study methods
relative survival are bound to annually presented life that are aimed at comparing country-based relative 5-
tables. These may also be calculated for shorter time year survival rates [32-35], because information of ab-
intervals. Inthis case theestimation of country-specific solute and relative stage distribution is linked to the
or regional life tables is recommended [31] for correspondingsurvivalrates.
Figure1ExtendedStandardizedScreeningIndex(eSSI)fornew Figure2RelativeOverallPerformanceIndex(ROPI)ofnew
malignantbreastcancercasesfromSEER-17(1990–2003), malignantbreastcancercasesfromSEER17(1990–2003),Norway
Norway(1969–2003)andSEER-17(2003)asareferenceobject. (1969–2003)andSEER-17(2003)asareferenceobject.
Jackeetal.BMCPublicHealth2013,13:34 Page6of9
http://www.biomedcentral.com/1471-2458/13/34
Conceptualizationandinterpretation fewer patients with metastases and unfavorable prog-
The conceptualization and epidemiological interpret- noses are included. However, the chance of survival
ation or health policy conclusions should be drawn in also increases in the later stages, because patients with
the country and time-specific context. Therefore, the in- metastases that are not apparent are detected earlier.
terpretation of the data examples remains restricted here Due to this effect also known as stage migration dis-
to a.) great variations (cross-sectional) between bench- torted stage-specific chances of survival result. The
mark objects and b.) convergence tendencies between overall chance of survival, however, is not affected by
theUSandNorwayoverthe courseoftime. thisphenomenon[42].
One contributing factor for this result is measured by On the contrary, comparison of SWI and STI facili-
the eSSI and SCI which are both conceptualized on the tates a greater understanding of the contribution of this
premise that the more effective early detection and stage migration effect to the overall outcome. For ex-
screening programs, the more the distribution of cases ample, if the stage-specific survival information (SWI) is
will be skewed toward the earlier stages of disease. This approximately 1.0 and the overall survival information
characteristic reflects common knowledge regarding the (STI) is substantially greater than 1.0, the improved
coherence between early detection programs and shifts overall outcome adjusted for the stage mix (SCI) appears
of stages [36-38] even if screening methods may be dis- to be primarily a treatment effect. On the other hand, if
cussed as somewhat controversial in certain age groups the SWI is substantially greater than 1.0 but the STI is
[39]. Also the conceptualization of the SWI captures this approximately 1.0, there is a stage migration occurring
stage migration effect directly. The SWI is based upon thatdoesnotappeartohaveamajorimpactonadminis-
the premisethat the more critical the work-up, the more tered treatments.
upstaging will occur and the better the survival at each Alongside, the comparison of survival conditions in
stage. However, unless the stage migration alters the time may also be distorted by the so-called lead-time
treatments administered, it will not impact the overall bias or zero-time shift [42,45]. Accordingly, screening-
survival, only the survival at each individual stage. The tests and diagnostic procedures can identify a disease
STI is based upon the premise that the better the over- even before the patient develops symptoms. This effect
all survival corrected for the case-mix, the more effect- leads to increasing survival times without actually lead-
ive the treatment being administered. The utilization of ingtoprolongationoflife,ifthehealth care effectiveness
each of these indices as a benchmark provides a means remains constant.
of identifying specific areas of program strength and
weakness. The combination of these indices to create Strengths
ROPI provides a benchmark for assessment of overall Compared to the original indices according to Beatty
program quality. et al. [6,7] an extended screening-index (eSSI) is
included, which is standardized in the same logic as all
Conceptionalpitfalls the other indices and which is expressed as a reciprocal.
The suggested method allows the comparison of coun- Therefore a positive association between the eSS-Index
tries, states, and districts on a longitudinal scale. This and the outcome-indicator is established. The latter has
perspective offers a high information grade in inter- been redefined by substituting the breast cancer-specific
nationalcomparisons.However,this means that anespe- fatality rates by relative survival rates. This approach has
cially high data quality is necessary that must fulfill the severaladvantages:
minimum requirements of representative, accurate,
complete,andcomparabledata[40,41].Asidefrom these (cid:4) Overallmore patientscanbeincludedinthe
formal requirements, some important statistical details analysesbecausesurvivalofall patientsisof
mustberegardedwhich arerelated tostagedistributions concern; regardlessofthe cause ofdeath. Thisleads
andsurvivalrates. toahigherstatisticalpower.
The comparison of the stage distribution may be (cid:4) Othercausesofdeathrespective frequently reported
distorted by the stage migration or the so called misleadinginformation [46,47]donothavetobe
“Will-Rogers-Phenomenon“[42-44]. According to this modeledinacompeting riskmodel following
phenomenon slow growing, “quiet”, and not appar- Gooley [8].
ently discernible disease symptoms such as metastases (cid:4) Background mortalityofthe populationcanbe
are discovered earlier due to increasingly powerful im- consideredinthe model.
aging procedures (diagnostic imaging). This means (cid:4) The calculationofthenon-parametricallyestimated
these cases are no longer classified as early disease relative survivalratesdoesnotrequire distribution
stages(0-II),butaslaterones(III-IV).Thus,thechance assumptions. They canbeestimated withexisting
ofsurvivalincreasesintheearlydiseasestages,because software solutionsappropriately.
Jackeetal.BMCPublicHealth2013,13:34 Page7of9
http://www.biomedcentral.com/1471-2458/13/34
(cid:4) Onestepforward,relative survivalratesmay be means that a comparison between “equivalent” compari-
standardizedbyvariables suchasage, gender, son objects should be achieved. Thus, homogenous
ethnicity etc.limitedonlybythe parameters benchmark objects in terms of “peer-groups” should be
availableinlifetables. identified [50]. This is recommended since most health
(cid:4) Benchmark-algorithmsand theoutcome-indicator care systems have evolved historically and, thus, infra-
relative survivalmay beextended toanyother structure characteristics and innovations can be imple-
diseaseaslong asaclassificationinsubsequent mented 1:1 from one country, region or district to
diseasephasesispossible. another under certain restrictions only [51,52]. However,
theidentificationof“peergroups”caneitherbebasedon
The outcome-indicator relative survival can also be content-relatedconsiderations(e.g.countrieswithnational
substituted by probabilities. For example, logistic regres- healthcareservices),onstatisticallychosendisease-specific
sionscan be calculatedtoquantifyreadmissionprobabil- parameters(e.g.distributionofrisk,prognosis,andpredict-
ities after inpatient treatment, if the corresponding ive factors) or both [9,53]. Overall incidence-based factors
benchmark parameter exists. Correspondingly, disease must be differentiated from patient-, disease-, and health
phases can be substituted by information on disease se- caresystem-centeredfactors[45]whichareresponsiblefor
verity (e.g. Charlson-score, Elixhauser-index [48,49]) as statisticaldistortionassociatedwithsurvivalanalyses.
long astheyhaveanordinal order.
Perspectives
Weakness Benchmark-algorithms that compare countries, states,
The proposed eSS-Index serves as a level-parameter and districts are highly complex and require great atten-
(“intercept”) which defines the general premise of the tion to research details [40,54]. To be able to take these
benchmark respective reference object. Its weakness is into consideration, high quality data is necessary
based on the arbitrary weights provided by the stage [9,53,55]. But high data quality in terms of accuracy and
numbers which devaluate the earliest disease identified completeness are hard to achieve. In case of the SEER
(stage 0) and emphasize the “not known” (stage 5, see register for example, some concerns have been docu-
Table 1). This arbitrariness might decrease the clinical mented [56] which should be thoroughly considered
valueof both,theeSSIandROPI.However,eSSIinforms when results are interpreted for health care decision
about stage distributions without any survival informa- making. This approach is even more recommended in-
tion andobtains thelogical consistency ofROPI. stead of the growing number of health care providers
TheconceptualpitfallsoftheWill-Rogerphenomenon, who seek to leave the data gathering process due to cost
stage migration and lead-time bias have already been reductions and missing benefits [57]. However,itisthese
mentioned (see above). The former is explored by com- datathatformthebasistoachieveahighertransparencyof
paring SWI- and STI, but lead-time bias would distort efficiency and health care quality which became a crucial
thatassessmentandcannotbedistinguishedfromanap- competition parameter in a growing health care industry.
parent increase in the incidence of the disease. In Thereforeitiscrucialforthenextstepinqualityassurance
addition, it is highly recommended from a methodo- to demonstrate how these data may achieve clues of evi-
logical point of view to estimate outcome-parameters dence for further improvements. The algorithms proposed
using the same method but with different regional life here may serve as first identifier of infrastructural differ-
tables. Therefore, population-based data is a crucial pre- encesinscreeningprogramsandcomparethesewithalter-
requisite, i.e. the catchment area of integrated networks nated consequences for the clinical work-up in countries,
or new organization forms in general should be clearly states and districts. However from a methodical point of
defined by their landmarks in order to obtain crucial view, the development of benchmark algorithms is not
demographic information from local or regional statis- complete. Corresponding tests are missing to generate
tical authorities. In addition, if possible and if these can- p-values for which distribution assumptions are requested.
notbeavoided,benchmarkobjectsshouldhavethesame If and under what circumstances certain distributions are
structural breaks in their nomenclature of disease phase given, will be the task of future developments. Finally, the
(e.g. AJCC, UICC). For example, the UICC Tumor- clinical meaning and interpretation of index differences
Node-Metastasis version 5 (1997–2001) was valid until between reference- and benchmark-objects has to be
version6(2002–2008)andversion7(since2009)became exploredinfutureapplications.
effective. The use of the same nomenclature should be
assuredfor referenceandbenchmark objects.But from a Conclusions
practice point of view, this is difficult to achieve due to To measure international, national, and regional health
time lags in implementation and documentation. Fur- care quality, the suggested algorithms and freely access-
thermore a fair benchmark has to be assured which ible SAS-macros BenchRelSurv and BenchRelSurvPlot
Jackeetal.BMCPublicHealth2013,13:34 Page8of9
http://www.biomedcentral.com/1471-2458/13/34
offer an additional tool to evaluate screening programs, Theauthorswishtothankthereviewersfortheirconstructivecomments
the clinical work-up and effectiveness in general. An ef- andsuggestions.
fectiveness comparison is sought that links the earliest
Authordetails
possibletimepointofaprogressivediseasewiththetime 1CentralInstituteofMentalHealth,MedicalFacultyMannheim/University
point of an absorbing result after the onset of the pri- Heidelberg,SquareJ5,68159Mannheim,Germany.2Departmentof
Gynecology,GynecologicalEndocrinologyandOncology,BreastCenter
marydisease consideringthebackgroundmortality(rela-
Regio,UniversityofMarburg,Marburg,Germany.
tive survival). This is especially relevant for diseases (e.g.
breast cancer) where the etiology and disease causes re- Received:27July2012Accepted:19December2012
Published:14January2013
main unclear. However, this concept can also be trans-
ferred to preventable diseases or avoidable mortalities
which have clearly defined disease courses (e.g. cardio- References
vascular disease, diabetes mellitus II) and which are 1. RosenbergRD,YankaskasBC,AbrahamLA,SicklesEA,LehmanCD,Geller
BM,CarneyPA,KerlikowskeK,BuistDSM,WeaverDL,BarlowWE,Ballard-
clearly avoidable by (behavioral) interventions. The soft-
BarbashR:Performancebenchmarksforscreeningmammography.
ware allows the identification of performance measure- Radiology2006,241:55–66.
ment in relation to comparative regions. It offers a first 2. SicklesEA,MigliorettiDL,Ballard-BarbashR,GellerBM,LeungJWT,
RosenbergRD,Smith-BindmanR,YankaskasBC:Performancebenchmarks
steptowardsanindepth researchanalysis. fordiagnosticmammography.Radiology2005,235:775–790.
3. PierceLJ,MoughanJ,WhiteJ,WinchesterDP,OwenJ,WilsonJF:1998–
1999patternsofcarestudyprocesssurveyofnationalpracticepatterns
Availability and requirements
usingbreast-conservingsurgeryandradiotherapyinthemanagement
Project name: Benchmarking relative survival (Bench- ofstageI-IIbreastcancer.IntJRadiatOncolBiolPhys2005,62:183–192.
RelSurv, BenchRelSurvPlot) 4. O'BrienMER,BorthwickA,RiggA,LearyA,AssersohnL,LastK,TanS,Milan
S,TaitD,SmithIE:Mortalitywithin30daysofchemotherapy:aclinical
Project home page: http://sourceforge.net/projects/
governancebenchmarkingissueforoncologypatients.BrJCancer2006,
benchrelsurv/ delivers files and examples as well as 95:1632–1636.
Technical ReportsinGerman andEnglish 5. KerbaM,MiaoQ,Zhang-SalomonsJ,MackillopW:Definingtheneedfor
breastcancerradiotherapyinthegeneralpopulation:acriterion-based
Operating system(s): Platform dependency of SAS 9.2 benchmarkingapproach.ClinOncol(RCollRadiol)2007,19:481–489.
orhigher 6. BeattyJ,ReesJ,AtwoodM,PuglieseM,BolejackV:Standardized
Programminglanguage:SAS9.2andhigher evaluationofregionalandinstitutionalbreastcanceroutcomes.http://
www.facs.org/cancer/coc/pdf/beatty_breast-outcomes.pdf?
Otherrequirements:None
bcsi_scan__58A52892D4FFF3EB=0&bcsi_scan_filename=beatty_breast-
License:None outcomes.pdf.
Any restrictions to use by non-academics: SAS 9.2 li- 7. BeattyJ,ReesJ,AtwoodM,PuglieseM,BolejackV:Standardized
evaluationofregionalandinstitutionalbreastcanceroutcomes.
cense orhigher AmJSurg2008,195:636–640.
8. GooleyTA,LeisenringW,CrowleyJ,StorerBA:Estimationoffailure
probabilitiesinthepresenceofcompetingrisks.Newrepresentationsof
Additional files
oldestimators.StatistMed1999,18:695–706.
9. JackeC,KalderM,KollerM,WagnerU,AlbertU:Systematicassessment
Additionalfile1:Example1.sas:Cross-sectionalbenchmarkfornew andimprovementofmedicaldataquality.Bundesgesundheitsbl-
malignantbreastcancercasesfromSEER-17(1999–2003)registers Gesundheitsforsch-Gesundheitsschutz2012,55:1495-1503.
withBenchRelSurvandBenchRelSurvPlot. 10. StausbergJ,NonnemacherM,WeilandD,AntonyG,NeuhäuserM:
ManagementofDataQuality.DevelopmentofaComputer-Mediated
Additionalfile2:Example2.sas:Longitudinalbenchmarkfornew
malignantbreastcancercasesfromSEER-17(1990–2003),Norway Guideline.StudHealthTechnolInform,2006:477–482.
(1969–2003)withBenchRelSurvandBenchRelSurvPlot. 11. NonnemacherM,WeilandD,StausbergJ:Datenqualitätindermedizinischen
Forschung:LeitliniezumadaptivenManagementvonDatenqualitätin
KohortenstudienundRegistern.Berlin:Med.-Wiss.Verl.-Ges;2007.
12. Surveillance,Epidemiology,andEndResults(SEER)Program(www.seer.
Competinginterests
cancer.gov)SEER*StatDatabase:Incidence-SEER18RegsResearchData+
Authorsdeclarenocompetinginterest.
HurricaneKatrinaImpactedLouisianaCases,Nov2011Sub(1973–2009
varying).LinkedToCountyAttributes-TotalU.S.,1969–2010Counties,National
Authors’contributions CancerInstitute,DCCPS,SurveillanceResearchProgram,SurveillanceSystems
COJcontributedsubstantiallytotheconception,acquisitionofdata,design Branch,releasedApril2012,basedontheNovember2011;submission.
andprogrammingofthesourcecode,analysisandinterpretationofdata, 13. CancerRegistryofNorway:CancerinNorway2008:CancerIncidence,
draftingthemanuscript,andgavefinalapproval.IRwasinvolvedinthe Mortality,SurvivalandPrevalenceinNorway.Oslo:CancerRegistryof
design,programming,andvalidationofthesourcecodeandrevisedand Norway;2009.
approvedthefinaldraft.USAwasinvolvedintheconception,analysis,and 14. TherneauT,GrambschP:ModelingSurvivalData:ExtendingtheCoxModel.
interpretationofdata,revisedcriticallyforimportantintellectualcontent,and NewYork:Springer;2001.
gaveapprovalforthefinaldraft.Allauthorsreadandapprovedthefinal 15. DickmanP,SloggettA,HillsM,HakulinenT:Regressionmodelsforrelative
manuscript. survival.StatMed2004,23:51–64.
16. PoharM,StareJ:Makingrelativesurvivalanalysisrelativelyeasy.Comput
Acknowledgement BiolMed2007,37:1741–1749.
ThisworkwassupportedbyagrantoftheGermanMinistryofHealth(Fö.Kz.: 17. HakulinenT:Cancersurvivalcorrectedforheterogeneityinpatient
FB2-43332-70/6)andtheMinistryofEducationandResearchwithinthe withdrawal.Biometrics1982,38:933–942.
program:Healthresearchtobenefitthepeople-Guidelineimplementation 18. HakulinenT,TenkanenL:Regressionanalysisofrelativesurvivalrates.
forearlydiagnosisandtreatmentofbreastcancer(Fö.-KZ:GFZPO1119302). JRStatSocSerCApplStat1987,36:309–317.
Jackeetal.BMCPublicHealth2013,13:34 Page9of9
http://www.biomedcentral.com/1471-2458/13/34
19. BrennerH:Long-termsurvivalratesofcancerpatientsachievedbythe 40. MainzJ,HjulsagerM,OgMTE,BurgaardJ:Nationalbenchmarking
endofthe20thcentury.Aperiodanalysis.Lancet2002,360:1131–1135. betweentheNordiccountriesonthequalityofcare.JSurgOncol2009,
20. BrennerH,GefellerO:Analternativeapproachtomonitoringcancer 99:505–507.
patientsurvival.Cancer1996,78:2004–2010. 41. MainzJ,BartelsP,RutbergH,KelleyE:Internationalbenchmarking.Option
21. BrennerH,HakulinenT,GefellerO:Computationalrealizationofperiod orillusion?IntJQualHealthCare2009,21:151–152.
analysisformonitoringcancerpatientsurvival.Epidemiology2002, 42. FeinsteinAR,SosinDM,WellsCK:TheWillRogersphenomenon.Stage
13:611–612. migrationandnewdiagnostictechniquesasasourceofmisleading
22. GeissK,MeyerM,Radespiel-TrögerM,GefellerO:SURVSOFT-Softwarefor statisticsforsurvivalincancer.NEnglJMed1985,312:1604–1608.
nonparametricsurvivalanalysis.ComputMethodsProgramsBiomed2009,96:63–71. 43. GolderWA:DasWill-Rogers-PhänomenundseineBedeutungfürdie
23. SariegoJ:Regionalvariationinbreastcancertreatmentthroughoutthe bildgebendeDiagnostik.Radiologe2009,49:348–354.
UnitedStates.AmJSurg2008,196:572–574. 44. SprattJS:WillRogersphenomenon.ArchSurg1992,127:868.
24. SariegoJ:PatternsofbreastcancerpresentationintheUnitedStates: 45. AutierP,BoniolM:Cautionneededforcountry-specificcancersurvival.
doesgeographymatter?AmSurg2009,75:545–9. Lancet2011,377:99–101.
25. BruckerSY,SchumacherC,SohnC,RezaiM,BambergM,WallwienerD: 46. ModelmogD,GoertchenR,SteinhardK,SinnHP,StahrH:Vergleichder
Benchmarkingthequalityofbreastcancercareinanationwide MortalitätsstatistikeinerStadtbeiunterschiedlicherObduktionsquote
voluntarysystem:thefirstfive-yearresults(2003–2007)fromGermany (GörlitzerStudie).Pathologe1991,12:191–195.
asaproofofconcept.BMCCancer2008,8:358. 47. SchelhaseT,WeberS:DieTodesursachenstatistikinDeutschland.
26. BruckerSY,WallwienerM,KreienbergR,JonatW,BeckmannMW,Bamberg ProblemeundPerspektiven.Bundesgesundheitsbl-Gesundheitsforsch-
M,WallwienerD,SouchonR:Optimizingthequalityofbreastcancercare Gesundheitsschutz2007,50:969–976.
atcertifiedgermanbreastcenters:abenchmarkinganalysisfor2003– 48. LieffersJR,BaracosVE,WingetM,FassbenderK:AcomparisonofCharlson
2009withaparticularfocusontheinterdisciplinaryspecialtyof andElixhausercomorbiditymeasurestopredictcolorectalcancer
radiationoncology.StrahlentherOnkol2011,187:89–99. survivalusingadministrativehealthdata.Cancer2011,117:1957–1965.
27. WallwienerM,BruckerSY,WallwienerD:Multidisciplinarybreastcentresin 49. LiB,EvansD,FarisP,DeanS,QuanH:Riskadjustmentperformanceof
Germany:areviewandupdateofqualityassurancethrough CharlsonandElixhausercomorbiditiesinICD-9andICD-10
benchmarkingandcertification.ArchGynecolObstet2012,285:1671–83. administrativedatabases.BMCHealthServRes2008,8:12.
28. DarouiP,GabelM,KhanAJ,HafftyBG,GoyalS:Utilizationofbreast 50. WalsheK:Internationalcomparisonsofthequalityofhealthcare.What
conservingtherapyinstages0,I,andIIbreastcancerpatientsinNew dotheytellus?QualSafHealthCare2003,12:4–5.
Jersey:anAmericanCollegeofSurgeonsNationalCancerDataBase 51. WalsheK:Understandingwhatworks-andwhy-inquality
(NCDB)analysis.AmJClinOncol2012,35:130–135. improvement:theneedfortheory-drivenevaluation.IntJQual
HealthCare2007,19:57–59.
29. NgW,DelaneyGP,JacobS,BartonMB:Estimationofanoptimal
52. NolteE,McKeeM:Measuringthehealthofnations.Analysisofmortality
chemotherapyutilisationrateforbreastcancer:settinganevidence-
amenabletohealthcare.BMJ2003,327:1129.
basedbenchmarkforthebest-qualitycancercare.EurJCancer2010,
46:703–712. 53. JackeC,KalderM,WagnerU,AlbertU:High-qualitydataforvalid
comparisonsanddecisions.Assessingtheaccuracyandcompletenessof
30. El-TamerMB,WardBM,SchifftnerT,NeumayerL,KhuriS,HendersonW:
medicaldata.BMCResearchNotes,inpress.
Morbidityandmortalityfollowingbreastcancersurgeryinwomen.
Nationalbenchmarksforstandardsofcare.AnnSurg2007,245:665–671. 54. MainzJ:Definingandclassifyingclinicalindicatorsforquality
31. DinkelR:DieBerechnungdesParameters„RelativeSurvival“fürein improvement.IntJQualHealthCare2003,15:523–530.
55. NonnemacherM,WeilandD,NeuhäuserM,StausbergJ:Adaptive
TumorregistermitregionalemEinzugsbereich.http://www.krebsregister.uni-
managementofdataqualityincohortstudiesandregisters.Proposalfor
rostock.de/docs/Relative%20Survival.pdf.
aguideline.ActaInformaticaMedica2007,15:225–230.
32. ColemanM,BabbP,DamieckiP,GrosclaudeP,HonjoS,JonesJ,KnererG,
56. BeattyJD,AdachiM,BonhamC,AtwoodM,PottsMS,HaftersonJL,AyeRW:
PitardA,QuinnM,SloggettA,deStavolaB:CancersurvivaltrendsinEngland
andWales1971–1995:deprivationandNHSRegion.London:TheStationery Utilizationofcancerregistrydataformonitoringqualityofcare.AmJ
Surg2011,201:645–649.
Office;1999[SeriesSMPS,vol.61].
57. GreeneFL,GilkersonS,TedderP,SmithK:Theroleofthehospitalregistry
33. ColemanMP,FormanD,BryantH,ButlerJ,RachetB,MaringeC,NurU,
inachievingoutcomebenchmarksincancercare.JSurgOncol2009,
TraceyE,CooryM,HatcherJ,McGahanCE,TurnerD,MarrettL,Gjerstorff
99:497–499.
ML,JohannesenTB,AdolfssonJ,LambeM,LawrenceG,MeechanD,Morris
EJ,MiddletonR,StewardJ,RichardsMA:CancersurvivalinAustralia,
Canada,Denmark,Norway,Sweden,andtheUK,1995–2007(the doi:10.1186/1471-2458-13-34
Citethisarticleas:Jackeetal.:Usingrelativesurvivalmeasuresfor
InternationalCancerBenchmarkingPartnership):ananalysisof
population-basedcancerregistrydata.Lancet2011,377:127–138. cross-sectionalandlongitudinalbenchmarksofcountries,states,and
districts:theBenchRelSurv-andBenchRelSurvPlot-macros.BMCPublic
34. ColemanM,QuaresmaM,BerrinoF,LutzJ,deAngelisR,CapocacciaR,Baili
Health201313:34.
P,RachetB,GattaG,HakulinenT,MicheliA,SantM,WeirH,ElwoodJ,
TsukumaH,KoifmanS,E-SilvaG,FrancisciS,SantaquilaniM,VerdecchiaA,
StormH,YoungJ:Cancersurvivalinfivecontinents:aworldwide
population-basedstudy(CONCORD).LancetOncol2008,9:730–756.
35. deBlacamC,GrayJ,BoyleT,KennedyMJ,HollywoodD,ButtJ,GriffinM,
NicholsonS,DunneB,WilsonG,McDermottR,MurphyP,ShortI,RowleyS,
ConnollyE,ReynoldsJV:Breastcanceroutcomesfollowinganational Submit your next manuscript to BioMed Central
initiativeinIrelandtorestructuredeliveryofservicesforsymptomatic and take full advantage of:
disease.Breast2008,17:412–417.
36. KatalinicA,BartelC,RaspeH,SchreerI:Beyondmammographyscreening.
• Convenient online submission
Qualityassuranceinbreastcancerdiagnosis(TheQuaMaDiproject).
BJC2007,96:157–161. • Thorough peer review
37. FerroniE,CamilloniL,JimenezB,FurnariG,BorgiaP,GuasticchiG,RossiPG:
• No space constraints or color figure charges
Howtoincreaseuptakeinoncologicscreening:asystematicreviewof
studiescomparingpopulation-basedscreeningprogramsand • Immediate publication on acceptance
spontaneousaccess.PrevMed2012,inpress. • Inclusion in PubMed, CAS, Scopus and Google Scholar
38. PulitiD,ZappaM:Breastcancerscreening:areweseeingthebenefit?
• Research which is freely available for redistribution
BMCMed2012,10:106.
39. AmirE,BedardPL,OcañaA,SerugaB:Benefitsandharmsofdetecting
clinicallyoccultbreastcancer.JNatlCancerInst2012,inpress. Submit your manuscript at
www.biomedcentral.com/submit