How long has NICE taken to produce Technology Appraisal time to

109 Pages·2012·2.16 MB·English

BMJ Open
B
M
J
O
p
e
n
: firs
t p
u
b
lis
h
e
d
a
How long has NICE taken to produce Technology Appraisal s
1
guidance? A retrospective study to estimate predictors of 0.1
1
time to guidance. 3
6
/b
m

jo
p
FJournal: BMJ Open en
-2
0
Manouscript ID: bmjopen-2012-001870 12
-0
r 0
Article Type: Research 1
8
7
0
Date Submitted by the Author: 26-Jul-2012 o
p n
1
Complete List of Authors: Casson, Steven; London School of Hygiene and Tropical Medicine, 1
Reuiz, Francis; National Institute for Health and Clinical Excellence, NICE J
a
International n
u
Mineers, Alec; London School of Hygiene and Tropical Medicine ary
r 2
<b>Primary Subject 0
Health econ omics 1
Heading</b>: 3
r . D
o
Health policy, Pharmacology and therapeutics, Public health, Evidence w
Secondary Subject Heading: n
based practice e lo
a
d
HEALTH ECONOMICvS, Health policy < HEALTH SERVICES ed
Keywords: ADMINISTRATION & MAiNAGEMENT, Protocols & guidelines < HEALTH fro
SERVICES ADMINISTRATION & MANAGEMENT, PUBLIC HEALTH, m
STATISTICS & RESEARCHe METHODS h
ttp
://b
w m
jo
p
e
n
.b
o m
j.c
o
n m
l o/
n
J
y an
u
a
ry
2
6
, 2
0
2
3
b
y
g
u
e
s
t. P
ro
te
c
te
d
b
y
c
o
p
y
rig
h
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml t.
Page 1 of 21 BMJ Open
B
1 M
J
2 O
p
3 e
n
45 : firs
6 How long has NICE taken to produce Technology Appraisal guidance? A t pu
b
7 retrospective study to estimate predictors of time to guidance lis
8 Steven G Casson MA(Cantab) MB BChir MSc MFPH1, Francis Ruiz BSc MSc2, Alec Miners BA MSc hed
9 a
s
1101 PhD3, 10.1
12 13
6
13 /b
m
14
1 SpR in Public Health, London School of Hygiene and Tropical Medicine, Keppel Street, London WC1E jo
15 p
16 F en
7HT, UK. -2
17 0
18 2 Senior Adviser (Heoalth Economics), NICE International, National Institute for Health and Clinical 12-0
19 r 0
1
20 Excellence, 71 High Holbor n, London WC1V 6NE 870
21 o
p n
22 3Lecturer in Health Economics, Faculty of Public Health and Policy, London School of Hygiene and 1
23 e 1 J
24 Tropical Medicine, Keppel Street, London WC1E 7HT, UK. an
u
25 e a
26 r ry 2
27 Address correspondence to: 01
3
28 [email protected] r . D
29 o
w
30 n
Contents e lo
31 a
d
32 v ed
33 • Word count of abstract, article summary, introiduction, methods, results, discussion and fro
34 references is 3317. m
35 • Three tables. e http
3367 • One figure w ://bm
38 • 11 references. jop
39 en
40 .b
o m
41 j.c
o
42 n m
43 l o/
n
44 J
45 y an
u
46 a
ry
47 2
6
48 , 2
49 0
2
50 3
b
51 y
g
52 u
e
5534 st. P
ro
55 te
c
56 te
d
57 b
58 y c
59 op
1 y
60 rig
h
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml t.
BMJ Open Page 2 of 21
B
1 M
J
2 O
p
3 e
Funding n
45 : firs
6 This research received no specific grant from any funding agency in the public, commercial or t pu
b
7 lis
8 not-for-profit sectors. It was conducted as part of an MSc degree at the LSHTM. he
d
9 a
s
10 1
11 Declaration of Competing Interests 0.1
12 13
6
13 SC has no relationship with NICE. AM is a current member of one of NICE’s Technology /b
m
14
jo
15 Appraisal Committee’s and its Technology Appraisals’ Decision Support Unit. FR is a member p
16 F en
-2
17 0
18 of NICE Internationoal, a not-for-profit consultancy service within NICE. FR, SC and AM have no 12-0
19 r 0
20 other non-financial intere sts that may be relevant to the submitted work. 187
0
21 o
p n
22 Ethics Statement 1
23 e 1 J
24 an
This project did not involve any human subjects or any human data. Therefore ethics approval u
25 e a
26 r ry 2
27 was not required. 01
3
28 r . D
29 o
w
30 Contributors n
e lo
31 a
d
32 SC collected, processed, and analysed the datva and drafted the paper. AM conceived the ed
33 i fro
34 m
subject for study, provided expert opinion on methodology / approach, contributed to the
35 e http
36 statistical analysis and helped write the text. FR contributed to the writing and discussion. AM ://b
37 w m
38 jo
39 is the guarantor. pen
40 .b
o m
41 j.c
Acknowledgements o
42 n m
43 l o/
44 Nina Pinwill, Associate Director at NICE, advised on appraisal processes and time point n J
45 y an
u
46 suitability for analysis, and provided the missing data described in the text. a
ry
47 2
6
48 , 2
49 Data sharing 0
2
50 3
b
51 No further data available. y g
52 u
e
5534 st. P
ro
55 te
c
56 te
d
57 b
58 y c
59 op
2 y
60 rig
h
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml t.
Page 3 of 21 BMJ Open
B
1 M
J
2 O
p
3 e
n
45 : firs
6 t pu
b
7 lis
8 he
Abstract d
9 a
s
10 1
0
11 Objectives: To assess how long the UK’s National Institute for Health and Clinical Excellence’s .1
12 13
6
13 (NICE) Technology Appraisal Programme has taken to produce guidance and to determine /b
m
14
jo
15 p
16 independent pFredictors of time to guidance. en
-2
17 0
18 o 12-0
19 r 0
1
20 Design: Retrospective s urvival analysis. 87
0
21 o
22 p n 1
23 e 1 J
24 Setting: Technology Appraisal’s guidance produced by NICE. an
u
25 e a
26 r ry 2
27 01
3
28 Datasource: All appraisals referred to NIrCE by February 2010 were included, except those . D
29 o
w
30 n
referred prior to 2001 and a number of thosee that were suspended. lo
31 a
d
32 v ed
33 i fro
34 m
35 Outcome measure: Duration from the start of an aeppraisal (when the scope document was http
36 ://b
37 released) until publication of guidance. w m
38 jo
p
39 en
40 .b
o m
41 Results: Single Technology Appraisals (STAs) were published significantly faster than Multiple j.c
o
42 n m
43 l o/
Technology Appraisals (MTAs) with median durations of 48.0 (interquartile range [IQR]; 44.3 to n
44 J
45 y an
46 75.4) and 74.0 (IQR; 60.9 to 114.0) weeks respectively (p<0.0001). Median time to publication ua
ry
47 2
48 exceeded published process timelines, even after adjusting for appeals. Results from the 6, 2
49 0
2
50 modelling suggest that STAs published results significantly faster guidance than MTAs after 3 b
51 y
g
52 u
adjusting for other covariates (by 36.2 weeks [95% CI -46.05 to -26.42 weeks]) and that appeals e
5534 st. P
against provisional guidance significantly increased the time to publication (by 42.83 weeks ro
55 te
c
56 te
[95% CI 35.50 to 50.17 weeks]). There was no evidence that STAs of cancer-related d
57 b
58 y c
59 op
3 y
60 rig
h
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml t.
BMJ Open Page 4 of 21
B
1 M
J
2 O
p
3 e
technologies took longer to complete compared with STAs of other technologies after adjusting n
45 : firs
6 for potentially confounding variables and only weak evidence suggesting that the time to t pu
b
7 lis
8 produce guidance is increasing each year (by 1.40 weeks [95% CI -0.35 to 2.94 weeks]). he
d
9 a
10 s 1
0
11 .1
12 Conclusions: The results from this study suggest that the STA process has resulted in 13
6
13 /b
m
14 significantly faster guidance compared with the MTA process irrespective of topic but that these jo
15 p
16 F en
gains are lost if appeals are made against provisional guidance. While NICE processes -2
17 0
18 continue to evolveo over time, a trade off might be that decisions take longer but at present there 12-0
19 r 0
1
20 87
is no evidence of a significant increase in duration. 0
21 o
p n
22 1
23 e 1 J
24 an
u
25 e a
26 r ry 2
27 Article summary 013
28 r . D
29 o
w
30 Article focus n
e lo
31 a
32 • How long has NICE’s Technology Appraisavls taken to produce guidance? ded
3334 • What features of an appraisal independently priedict the time to publication of guidance? from
3356 Key messages e http://b
37 w m
38 • The STA process has reduced the time to publication by about 36 weeks irrespective of topic. jo
p
39 en
40 • Appeals against final appraisal determinations have more than doubled the time it takes for STAs .b
o m
41 j.c
42 to conclude. No other factors were strongly predictive of the timne to guidance. om
43 l o/
44 • No variables predicting the likelihood of an appeal were identified. n J
45 y an
u
46 Strengths and limitations of this study a
ry
47 2
48 • Use of survival analysis is a significant improvement on previous studies addressing the primary 6, 2
49 0
2
50 question. 3
b
51 y
52 • Time to guidance is not in itself an indicator of the ‘quality’ of decision. gu
e
5534 • Other factors might also independently predict the time to guidance, such as consideration of st. P
ro
55 te
56 patient access schemes and the number of consultees on each appraisal. cte
d
57 b
58 y c
59 op
4 y
60 rig
h
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml t.
Page 5 of 21 BMJ Open
B
1 M
J
2 O
p
3 e
Introduction n
45 : firs
6 In England and Wales, the primary role of National Institute for Health and Clinical Excellence’s t pu
b
7 lis
8 (NICE’s) Centre for Health Technology Evaluation is to produce guidance on the appropriate he
d
9 a
s
10 use of technologies for the NHS. Prior to 2005 all appraisals were undertaken using its Multiple 1
0
11 .1
12 Technology Appraisal (MTA) process [1]. However, following criticism of the slow production of 136
13 /b
m
14
guidance [2], [3], NICE established the Single Technology Appraisal (STA) process in 2005 with jo
15 p
16 F en
-2
17 the objective of producing faster guidance closer to the time of product launch [4, 5]. Precise 0
18 o 12-0
19 details of both processres can be found elsewhere [1, 6] but the STA is largely similar to the 0
1
20 87
0
21 MTA but focuses on a single technology rather than a broader set, as its name implies, and the o
p n
22 1
23 independent assessment of thee evidence is restricted to a critique of the manufacturers 1 J
24 an
u
2256 submission rather than including ae der novo systematic review and economic evaluation; it has ary 2
27 been likened to the process used by the Scottish Medicine’s Consortium [8]. STA adoption has 013
28 r . D
29 o
w
30 been rapid, increasing from 13% of all technology appraisals in June 2008 to 43.4% by n
e lo
31 a
d
32 February 2010. v ed
33 i fro
34 m
35 STAs and MTAs should in theory take 43 and 60 weeeks respectively to conclude in the absence http
36 ://b
37 of an appeal against the provisional guidance (more forwmally known as a ‘final appraisal m
38 jo
p
39 determination’). A number of studies have attempted to assess whether the processes have en
40 .b
o m
41 met these targets and whether the STA process has resulted in faster guidance [7-9]. For j.co
42 n m
43 l o/
example, Ford et al suggests that the STA has reduced the time to produce guidance, but not n
44 J
45 y an
u
46 for cancer-related technologies [8]. O’Neil also suggests that the STA has reduced the average a
ry
47 2
6
48 time to guidance, by approximately 1 year [9]. However, both analyses are limited. First, Ford , 2
49 0
2
50 only considers the time from product launch to guidance, rather than from the time NICE is 3 b
51 y
g
52 u
formally requested to appraise a technology by the Department of Health; NICE only ‘controls’ e
5534 st. P
the latter to some extent. Second, the studies only include completed appraisals; no ro
55 te
c
56 te
adjustments were made for ongoing, and potentially lengthy, assessments meaning the results d
57 b
58 y c
59 op
5 y
60 rig
h
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml t.
BMJ Open Page 6 of 21
B
1 M
J
2 O
p
3 e
could be biased. Last, while Ford and O’Neill attempted to identify independent predictors of n
45 : firs
6 the time to guidance, none assessed these using formal statistical approaches for time to event t pu
b
7 lis
8 data and no attempts were made to formally identify the individual contribution of each he
d
9 a
10 explanatory variable to the total time. The purpose of this study is to address all of these s 1
0
11 .1
12 issues. 13
6
13 /b
m
14
jo
15 p
16 F en
-2
17 0
18 o 12-0
19 r 0
1
20 87
0
21 o
p n
22 1
23 e 1 J
24 an
u
25 e a
26 r ry 2
27 01
3
28 r . D
29 o
w
30 n
e lo
31 a
d
32 v ed
33 i fro
34 m
35 e http
36 ://b
37 w m
38 jo
p
39 en
40 .b
o m
41 j.c
o
42 n m
43 l o/
n
44 J
45 y an
u
46 a
ry
47 2
6
48 , 2
49 0
2
50 3
b
51 y
g
52 u
e
5534 st. P
ro
55 te
c
56 te
d
57 b
58 y c
59 op
6 y
60 rig
h
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml t.
Page 7 of 21 BMJ Open
B
1 M
J
2 O
p
3 e
Methods n
45 : firs
6 t pu
7 Inclusion criteria blis
8 he
d
9 All appraisals referred to NICE by Feb 2010 were considered for inclusion. However, MTAs a
s
10 1
0
11 prior to 2001 were excluded as they followed a different process to more recent MTAs . .1
12 13
6
13 Appraisals were also excluded (19 STAs and 7 MTAs) if they had been suspended or /b
m
14
jo
15 p
16 postponed folloFwing initial referral from the Department of Health but before NICE issued the en
-2
17 0
18 final scope documoent. For example, the Institute was asked to assess the cost-effectiveness of 12-0
19 r 0
1
20 ‘faller’s clinics’ for elderly individuals. However, the appraisal was stopped before consultees 87
0
21 o
p n
22 were asked to submit evidence as the difficulties of defining an intervention became clear. 1
23 e 1 J
24 an
u
25 e a
26 r ry 2
27 Key dates, durations and data sources 01
3
28 r . D
29 o
Data for the analysis was taken from NICE’s website. A small amount of missing data w
30 n
e lo
31 a
(comprised of 21 start dates, 6 suspension dates, 4 appeal announcement dates, and 6 process d
32 v ed
33 i fro
34 types i.e. MTA or STA) was provided directly by NICE on request. The ‘core’ appraisal time m
35 e http
36 period was bounded as follows. Start dates were calculated for the majority of appraisals using ://b
37 w m
38 the ‘final scope’ date, as this was the earliest consistently-re corded time point available jop
39 en
40 .b
throughout the whole dataset; this is also in line with when NICoE ‘starts the clock’. Scope m
41 j.c
o
42 documents include information on the intervention(s) to be evaluanted and the relevant m
43 l o/
n
44 J
45 comparator programmes, and can be viewed as a formal appraisal staryt date for the purposes of an
u
46 a
47 inviting and constructing evidenced based submissions. Where this date was unknown (for ry 2
6
48 , 2
49 1 STA and 6 MTAs), the start date was inferred using the ‘closing date for submissions to 0
2
50 3
b
51 appraisal process by consultees’. This time point is scheduled to occur at week 9 in the STA y g
52 u
e
5534 process or week 14 for a MTA. Subtraction of the relevant number of weeks (9 or 14) allowed st. P
ro
55 the start of the core process to be inferred. tec
56 te
d
57 b
58 y c
59 op
7 y
60 rig
h
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml t.
BMJ Open Page 8 of 21
B
1 M
J
2 O
p
3 e
Statistical analysis n
45 : firs
6 The data effectively represent ‘to time to an event’, meaning it was analysed using survival t pu
b
7 lis
8 analysis techniques with the ‘event’ being publication of guidance. Time to publication was he
d
9 a
10 initially assessed using Kaplan-Meier (KM) techniques, stratified by the parameter of interest s 1
0
11 .1
12 (eg. STA / MTA process). Statistical significance was estimated using the log-rank test. The 13
6
13 /b
m
14 end (censor) date was taken to be the date final guidance was published, the date an appraisal jo
15 p
16 was suspendeFd or 13th February 2010, whichever occurred first. Rather than use Cox en-2
17 0
1189 proportional hazarod mrodels to adjust KM results for multiple independent parameters, 12-00
1
20 87
parametric techniques were instead used. This was because the latter is able to generate 0
21 o
p n
22 1
23 predictions of time to publicatieon of guidance for censored events and to provide direct 1 J
24 an
u
25 estimations of the independent coentribution of each predictive variable to the total time to a
26 r ry 2
27 guidance (ie. the marginal effect). For e xample, the number of weeks an appeal has added to 01
3
28 r . D
29 o
the length of a MTA or STA can be calculated, all other factors held constant. A number of w
30 n
e lo
31 a
different parametric survival models were fitted to the data including exponential, Weibull, d
32 v ed
3334 lognormal, loglogistic, Gompertz and gamma. Thei model that minimised Akaike’s information from
3356 criterion (AIC) was selected for use. Sensitivity anaelysis was also used to assess the effect of http://b
37 w m
38 using alternative parameteric model forms. Additionally, log istic regression was used to identify jop
39 en
40 assess whether a number of independent variables predicted othe likelihood of an appeal. The .bm
41 j.c
o
42 proportion of appraisals completing within anticipated process timnes (43 and 60 weeks for STAs m
43 l o/
n
44 J
45 and MTAs) were assessed by assuming a binomial distribution. All anaylyses were undertaken an
u
46 a
using STATA v12. ry
47 2
6
48 , 2
49 0
2
50 3
b
51 y
g
52 Choice of independent variables u
e
5534 The choice of appraisal process (STA or MTA) was an obvious parameter for inclusion, since st. Pro
55 te
c
56 STAs are designed to be shorter than MTAs. Other parameters were identified using existing te
d
57 b
58 y c
59 op
8 y
60 rig
h
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml t.
Page 9 of 21 BMJ Open
B
1 M
J
2 O
p
3 e
literature and consideration of the underlying processes. For example, it is logical that an n
45 : firs
6 appeal against provisional guidance could add substantially to the time it takes to publish final t pu
b
7 lis
8 guidance. Other authors have also suggested that cancer appraisals are typically more he
d
9 a
10 complex and ‘controversial’, given that they tend to be associated with high incremental cost- s 1
0
11 .1
12 effectiveness ratios, meaning they take longer to complete. NICE considers revising published 13
6
13 /b
m
14 appraisal guidance every 1-3 years. Given that in theory these revisions should be adding to an jo
15 p
16 F en
existing evidence base, it was suspected that they might take a shorter time to complete -2
17 0
18 compared with othoer appraisals. O’Neill suggested that there was no evidence that appraisals 12-0
19 r 0
1
20 87
are generally taking longer to complete, a so called ‘time-trend’. However, they also suggest 0
21 o
p n
22 1
23 that this conclusion should be erevisited [using more formal statistical approaches]. 1 J
24 an
u
25 e a
26 r ry 2
27 For these reasons, the following indepe ndent variables were included in the survival analysis 01
3
28 r . D
29 o
and logistic regression analysis: review of existing appraisal (yes / no), drug (yes / no), cancer- w
30 n
e lo
31 a
related topic (yes / no), whether an appeal on the final appraisal determination (yes / no), d
32 v ed
3334 calendar year of appraisal start (2001 to 2010) andi an interaction term between STA and cancer from
3356 to test whether there was a difference between canceer-related and remaining STAs. http://b
37 w m
38 Other parameters were considered for inclusion, some of which had previously been studied, jo
p
39 en
40 such as consideration of patient access schemes, guidance thoat ultimately restricted the use of .bm
41 j.c
o
42 a technology and the number of groups (consultees) who were fornmally engaged with an m
43 l o/
n
4445 appraisal. However, they were ultimately rejected from the final model ybecause of difficulties in Jan
u
46 a
consistently collecting this evidence. For example, a number of patient access schemes have ry
47 2
6
48 , 2
49 been submitted to NICE, but only more recently has this become a formal part of NICE’s 0
2
50 3
b
51 appraisal processes. y
g
52 u
e
5534 The basic tested hypothesis was that none of the independent parameters independently st. P
ro
55 predicted the time to publication of guidance. te
c
56 te
d
57 b
58 y c
59 op
9 y
60 rig
h
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml t.