Evaluating and Optimising the Retrieval of Research Evidence for Systematic Reviews of Adverse ...

457 Pages·2013·4.67 MB·English

by  Su Golder

Evaluating and Optimising the
Retrieval of Research Evidence
for Systematic Reviews of
Adverse Drug Effects and
Adverse Drug Reactions

A Thesis Submitted for the Degree
of Doctor of Philosophy

By

Susan Pamela Golder

Department of Health Sciences,
University of York

July 2013
Abstract
Systematic reviews can provide timely, reliable evidence on which to make informed
decisions. In order to make balanced decisions, information is not only needed on
the benefits of an intervention, but also on its adverse effects. Yet few systematic
reviews incorporate adverse effects data in their analysis. There is currently a lack
of guidance on how to identify adverse effects data, this may impede systematic
reviewers. This thesis helps address this situation by evaluating and optimising the
methods for retrieval of research evidence for systematic reviews of adverse effects.

The first stage of this programme of research critically reviews the methodological
literature relating to the retrieval and inclusion of adverse effects data, including
aspects such as the impact of study design (for example RCTs and cohort studies),
database search strategies (for example in MEDLINE and EMBASE), sources of
data (including database and non-database sources), publication status and funding
status.

Second, the results of a survey of the literature searching methods used in 849
systematic reviews of adverse effects are presented. Data were collated on aspects
such as sources searched, search strategy design and the standard of reporting of
the methods used. The reviews are published over a 17 year time period (1994-
2011) thus enabling time trends analysis. The methods used in these systematic
reviews of adverse effects are also compared with those reported in surveys of
other types of reviews.

Further potentially relevant evidence is incorporated to address gaps identified in
the literature. A detailed analysis is provided of the contribution of different sources
of data for adverse drug reactions using 58 included studies from a case study
systematic review. The same case study systematic review is then used to measure
the performance of adverse effects search filters in MEDLINE and EMBASE.

Finally 242 included papers from a series of 26 systematic reviews are evaluated to
strengthen the evidence base regarding adverse effects search filters and to assess
individual adverse effects search terms in MEDLINE, EMBASE, and Science
Citation Index (SCI).

The strengths and weaknesses of the analyses are discussed and implications for
practice and guidance presented along with recommendations for future research.
2
List of contents
List of tables .......................................................................................................... 9
List of figures ...................................................................................................... 12
Acknowledgements ............................................................................................ 14
Dedication............................................................................................................ 14
Publications ......................................................................................................... 15
Conference presentations ................................................................................. 17
Other presentations ........................................................................................... 19
Invited workshops .............................................................................................. 20
Author’s declaration ........................................................................................... 21
Chapter 1 Introduction ................................................................................... 22
1.1 Background ........................................................................................... 22
1.2 Systematic reviews and adverse effects .......................................... 23
1.3 Aims and objectives of the thesis ..................................................... 24
1.4 Research questions .............................................................................. 25
1.5 Thesis structure .................................................................................... 25
1.6 Summary ............................................................................................... 27
Chapter 2 Incorporating adverse effects into systematic reviews .......... 28
2.1 What are adverse effects? .................................................................. 28
2.2 Why are adverse drug reactions important? ................................... 30
2.3 What is known about adverse drug reactions pre-licensing? ....... 33
2.4 Why are systematic reviews incorporating adverse effects
needed? ........................................................................................................... 33
2.5 Do systematic reviews incorporate adverse effects? ..................... 36
2.6 What are the challenges of retrieving information on adverse
effects? ............................................................................................................. 37
2.7 Summary ............................................................................................... 40
Chapter 3 Methodological overview of the literature relating to the
retrieval of information on adverse effects ................................................... 41
3
3.1 Introduction .......................................................................................... 41
3.2 Methods ................................................................................................. 42
3.3 Results ................................................................................................... 45
3.4 Summary ............................................................................................... 48
Chapter 4 Section A of the methodological overview: different study
designs for information on adverse effects ................................................... 49
4.1 Introduction .......................................................................................... 49
4.2 Methods ................................................................................................. 57
4.3 Results ................................................................................................... 60
4.4 Discussion ............................................................................................. 78
4.5 Limitations ............................................................................................. 81
4.6 Conclusions ........................................................................................... 83
4.7 Summary ............................................................................................... 83
Chapter 5 Section B of the methodological overview: sources of
information on adverse effects ........................................................................ 84
5.1 Introduction .......................................................................................... 84
5.2 Methods ................................................................................................. 85
5.3 Results ................................................................................................... 86
5.4 Discussion ............................................................................................. 95
5.5 Limitations ............................................................................................. 95
5.6 Conclusions ........................................................................................... 98
5.7 Summary ............................................................................................... 98
Chapter 6 Section C of the methodological overview: database
search strategies for information on adverse effects .................................. 99
6.1 Introduction .......................................................................................... 99
6.2 Methods ............................................................................................... 100
6.3 Results ................................................................................................. 102
6.4 Discussion ........................................................................................... 105
4
6.5 Limitations ........................................................................................... 106
6.6 Conclusions ......................................................................................... 107
6.7 Summary ............................................................................................. 107
Chapter 7 Section D of the methodological overview: impact of
publication status on the reporting of adverse effects .............................. 108
7.1 Introduction ........................................................................................ 108
7.2 Methods ............................................................................................... 109
7.3 Results ................................................................................................. 111
7.4 Discussion ........................................................................................... 117
7.5 Limitations ........................................................................................... 119
7.6 Conclusions ......................................................................................... 120
7.7 Summary ............................................................................................. 121
Chapter 8 Section E of the methodological overview: impact of
funding source on the reporting of adverse effects ................................... 122
8.1 Introduction ........................................................................................ 122
8.2 Methods ............................................................................................... 123
8.3 Results ................................................................................................. 124
8.4 Discussion ........................................................................................... 128
8.5 Limitations ........................................................................................... 129
8.6 Conclusions ......................................................................................... 130
8.7 Summary ............................................................................................. 131
Chapter 9 Section F of the methodological overview: other issues
related to the retrieval of information on adverse effects ........................ 132
9.1 Introduction ........................................................................................ 132
9.2 Methods ............................................................................................... 132
9.3 Results ................................................................................................. 133
9.4 Discussion ........................................................................................... 135
9.5 Limitations ........................................................................................... 136
5
9.6 Conclusions ......................................................................................... 136
9.7 Summary ............................................................................................. 136
Chapter 10 Methods used to search for adverse effects data in
systematic reviews: 1994 to 2011 ................................................................ 137
10.1 Introduction ..................................................................................... 137
10.2 Methods ............................................................................................ 138
10.3 Results .............................................................................................. 140
10.4 Discussion ........................................................................................ 156
10.5 Limitations ....................................................................................... 167
10.6 Conclusions ...................................................................................... 168
10.7 Summary ......................................................................................... 169
Chapter 11 The contribution of different sources for information on
adverse drug reactions: a case study of fractures with
thiazolinediones ................................................................................................ 170
11.1 Introduction ..................................................................................... 170
11.2 Methods ............................................................................................ 171
11.3 Results .............................................................................................. 174
11.4 Discussion ........................................................................................ 187
11.5 Limitations ....................................................................................... 189
11.6 Conclusions ...................................................................................... 190
11.7 Summary ......................................................................................... 191
Chapter 12 Search filters in MEDLINE and EMBASE: a case study of
fractures with thiazolidinediones ................................................................... 192
12.1 Introduction ..................................................................................... 192
12.2 Methods ............................................................................................ 193
12.3 Results .............................................................................................. 194
12.4 Discussion ........................................................................................ 197
12.5 Limitations ....................................................................................... 198
12.6 Conclusions ...................................................................................... 198
6
12.7 Summary ......................................................................................... 199
Chapter 13 Adverse effects terms in database records: an analysis of
the included papers from 26 systematic reviews ....................................... 200
13.1 Introduction ..................................................................................... 200
13.2 Methods ............................................................................................ 201
13.3 Results .............................................................................................. 205
13.4 Discussion ........................................................................................ 211
13.5 Limitations ....................................................................................... 214
13.6 Conclusions ...................................................................................... 214
13.7 Summary ......................................................................................... 215
Chapter 14 Discussion .................................................................................... 216
14.1 Overall summary ............................................................................ 216
14.2 Main Findings .................................................................................. 217
14.3 Implications for practice ................................................................ 219
14.4 Dissemination and implications for guidance ............................ 222
14.5 Implications for research .............................................................. 224
14.6 Conclusions ...................................................................................... 226
Chapter 15 Appendices ................................................................................... 228
Appendix A: Search strategies for methodological overviews in
Chapters 3 to 9 ................................................................................................. 228
Appendix B: Tables and figures for methodological overviews in
Chapters 4 to 9 ................................................................................................. 245
Appendix C: Tables for overview of methods used to search for
adverse effects data in systematic reviews in Chapter 10 ....................... 355
Appendix D: Protocol for case study systematic review evaluating
fractures with rosiglitazone and pioglitazone for Chapters 11 and 12 ... 362
Appendix E: Search strategies for case study systematic review in
Chapters 11 and 12:........................................................................................ 368
7
Appendix F: Included and excluded studies for case study systematic
review in Chapters 11 and 12 ........................................................................ 394
Appendix G: MEDLINE and EMBASE searches tested in Chapter 12 ....... 406
Appendix H: Published adverse effects search filters for MEDLINE and
EMBASE for Chapters 12 and 13 ................................................................... 410
Appendix I: Adverse effects terms in database records in Chapter 13 .. 412
Appendix J: Sensitivity of searches in case study systematic review
and selection of databases in systematic reviews of adverse effects. .... 420
Abbreviations .................................................................................................... 421
References ......................................................................................................... 422

8
List of tables
Table 3.1 Sources searched for included studies ......................................... 43
Table 4.1 Confidence interval overlap between study designs in studies
measuring incidence .......................................................................................... 68
Table 4.2 Confidence interval overlap and agreement between study
designs in studies measuring risk ratios or odds ratios .............................. 69
Table 4.3 Ratio of odds ratios (RORs) of adverse effects in study design
comparisons ........................................................................................................ 72
Table 4.4 Pooled ratio of incidence of adverse effects in study design
comparisons ........................................................................................................ 74
Table 5.1 Sources compared in the 19 included methodological
evaluations .......................................................................................................... 88
Table 7.1 Observed and estimated estimates of risk ratios with
confidence intervals ......................................................................................... 114
Table 7.2 Relative risks/odds ratios and confidence intervals for
unpublished studies, published studies, and published and unpublished
studies combined ............................................................................................. 116
Table 10.1 Sources searched in order of frequency .................................. 145
Table 10.2 Categories of search terms used in database search strategies
............................................................................................................................ 152
Table 10.3 Profession of searcher and number of sources searched ...... 155
Table 10.4 Sources searched in systematic reviews of adverse effects
compared to other reviews ............................................................................. 159
Table 10.5 Reporting of items in systematic reviews ................................ 165
Table 11.1: References retrieved by databases, in order of sensitivity . 176
Table 11.2 References (RCTs and observational studies) retrieved by
non-bibliographic databases .......................................................................... 177
Table 11.3 RCTs retrieved by databases, in order of sensitivity ............. 182
Table 11.4 Observational studies retrieved by databases, in order of
sensitivity .......................................................................................................... 184
Table 11.5 Marginal sensitivity, marginal precision and additional number
needed to read using the source with the highest number of relevant
records first ....................................................................................................... 185
Table 11.6 Marginal sensitivity, marginal precision and additional number
needed to read using order of sources in current practice ....................... 186
9
Table 12.1 Sensitivity, precision, and number needed to read (NNR)
using search filters in MEDLINE and EMBASE ............................................. 196
Table 13.1 Average percentage of records with adverse effects terms in
the title, abstract or indexing in the present study and in Derry et al
2001 ................................................................................................................... 206
Table 13.2 Adverse effects terms in the title, abstract or indexing of
records in MEDLINE, EMBASE or Science Citation Index (SCI) ............... 208
Table 13.3 Sensitivity of MEDLINE and EMBASE search strategies for
adverse effects ................................................................................................. 210
Table 15.1 Characteristics of included studies in Chapter 4 .................... 245
Table 15.2 Excluded studies in Chapter 4 ................................................... 291
Table 15.3 Data sources for information on adverse effects .................... 306
Table 15.4 Characteristics of included studies for Chapter 5 ................... 311
Table 15.5 Excluded studies in Chapter 5 ................................................... 329
Table 15.6 Characteristics of included studies in Chapter 6 .................... 332
Table 15.7 Excluded studies in Chapter 6 ................................................... 336
Table 15.8 Methodological quality of included studies in Chapter 6 ....... 337
Table 15.9 Characteristics of included studies in Chapter 7 .................... 339
Table 15.10 Excluded studies in Chapter 7 ................................................. 345
Table 15.11 Characteristics of included studies in Chapter 8 .................. 346
Table 15.12 Excluded studies in Chapter 8 ................................................. 350
Table 15.13 Characteristics of included studies in Chapter 9 .................. 351
Table 15.14 Systematic reviews by type of intervention 1994 to 2011 . 355
Table 15.15 Systematic reviews by types of study designs included 1994-
2011 ................................................................................................................... 356
Table 15.16 Databases and other sources searched in systematic reviews
of adverse effects 1994-2011 ........................................................................ 357
Table 15.17 Systematic reviews searching the top four databases 1994-
2011 ................................................................................................................... 358
Table 15.18 Systematic reviews searching the top four non-database
sources 1994-2011 .......................................................................................... 359
Table 15.19 Systematic reviews with reproducible search strategies
1994-2011 ......................................................................................................... 360
Table 15.20 Systematic reviews with date or language restrictions 1994-
2011 ................................................................................................................... 361
10