Methods Guide for Comparative Effectiveness Reviews: Integrating Bodies of Evidence: Existing Systematic Reviews and Primary Studies

2015·0.22 MB·English

Methods Guide
for Comparative Effectiveness Reviews
Integrating Bodies of Evidence: Existing Systematic
Reviews and Primary Studies
This report is based on research conducted by the Agency for Healthcare Research and Quality
(AHRQ) Evidence-based Practice Centers’ 2014 Methods Workgroup 3. The findings and
conclusions in this document are those of the authors, who are responsible for its contents; the
findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no
statement in this report should be construed as an official position of AHRQ or of the U.S.
Department of Health and Human Services.
The information in this report is intended to help health care decisionmakers—patients and
clinicians, health system leaders, and policymakers, among others—make well-informed
decisions and thereby improve the quality of health care services. This report is not intended to
be a substitute for the application of clinical judgment. Anyone who makes decisions concerning
the provision of clinical care should consider this report in the same way as any medical
reference and in conjunction with all other pertinent information (i.e., in the context of available
resources and circumstances presented by individual patients).
AHRQ or U.S. Department of Health and Human Services endorsement of any derivative
products that may be developed from this report, such as clinical practice guidelines, other
quality enhancement tools, or reimbursement or coverage policies may not be stated or implied .
This document is in the public domain and may be used and reprinted without permission except
those copyrighted materials that are clearly noted in the document. Further reproduction of those
copyrighted materials is prohibited without the specific permission of copyright holders.
This research was funded through contracts from the Agency for Healthcare Research and
Quality to the following Evidence-based Practice Centers: The Johns Hopkins University (290-
2012-00007-I), Pacific Northwest (290-2012-00014-I), University of Alberta (290-2012-00013-
I), University of Minnesota (290-2012-00016-I), RAND Corporation (290-2012-0006-I), RTI
International and the University of North Carolina (290-2012-00008-I), and the Scientific
Resource Center for the EPC Program (290-2012-00004-C).
Persons using assistive technology may not be able to fully access information in this report. For
assistance, contact [email protected]
None of the investigators have any affiliations or financial involvement that conflicts with the
material presented in this report.
Suggested citation: Robinson KA, Chou R, Berkman ND, Newberry SJ, Fu R, Hartling L,
Dryden D, Butler M, Foisy M, Anderson J, Motu’apuaka ML, Relevo R, Guise JM, Chang S.
Integrating Bodies of Evidence: Existing Systematic Reviews and Primary Studies. Methods
Guide for Comparative Effectiveness Reviews (Prepared by the Scientific Resource Center under
Contract No. 290-2012-00004-C). AHRQ Publication No. 15-EHC007-EF. Rockville, MD:
Agency for Healthcare Research and Quality. February 2015.
www.effectivehealthcare.ahrq.gov/reports/final.cfm.
ii
Prepared by:
Scientific Resource Center
Portland, OR

Investigators:
Karen A. Robinson, Ph.D.1
Roger Chou, M.D.2
Nancy D. Berkman, Ph.D.3
Sydne J. Newberry, Ph.D.4
Rongwei Fu, Ph.D.5
Lisa Hartling, B.Sc.P.T., M.Sc., Ph.D.6
Donna Dryden, Ph.D.6
Mary Butler, Ph.D.7
Michelle Foisy, B.A.-Hon, M.A.6
Johanna Anderson, M.P.H.5
Makalapua Motu’apuaka, B.S.5
Rose Relevo, M.L.I.S., M.S.5
Jeanne-Marie Guise, M.D., M.P.H.5
Stephanie Chang, M.D., M.P.H.8

1
Johns Hopkins University Evidence-based Practice Center, Baltimore, MD
2
Pacific Northwest Evidence-based Practice Center, Portland, OR
3
RTI-University of North Carolina Evidence-based Practice Center, Research Triangle Park, NC
4
Southern California Evidence-based Practice Center, RAND, Santa Monica, CA
5
Scientific Resource Center for the AHRQ Effective Health Care Program, Portland VA
Research Foundation, VA Portland Health Care Systems, Portland, OR
6
Department of Pediatrics, University of Alberta, Edmonton, Canada
7
Minnesota Evidence-based Practice Center, Minneapolis, MN
8
Center for Evidence and Practice Improvement, Agency for Healthcare Research and Quality,
Rockville, MD
iii
Preface
The Agency for Healthcare Research and Quality (AHRQ), through its Evidence-based
Practice Centers (EPCs), sponsors the development of evidence reports and technology
assessments to assist public- and private-sector organizations in their efforts to improve the
quality of health care in the United States. The reports and assessments provide organizations
with comprehensive, science-based information on common, costly medical conditions and new
health care technologies and strategies. The EPCs systematically review the relevant scientific
literature on topics assigned to them by AHRQ and conduct additional analyses when
appropriate prior to developing their reports and assessments.
Strong methodological approaches to systematic review improve the transparency,
consistency, and scientific rigor of these reports. Through a collaborative effort of the Effective
Health Care (EHC) Program, the Agency for Healthcare Research and Quality (AHRQ), the
EHC Program Scientific Resource Center, and the AHRQ Evidence-based Practice Centers have
developed a Methods Guide for Comparative Effectiveness Reviews. This Guide presents issues
key to the development of Systematic Reviews and describes recommended approaches for
addressing difficult, frequently encountered methodological issues.
The Methods Guide for Comparative Effectiveness Reviews is a living document, and will be
updated as further empiric evidence develops and our understanding of better methods improves.
We welcome comments on this Methods Guide paper. They may be sent by mail to the Task
Order Officer named below at: Agency for Healthcare Research and Quality, 540 Gaither Road,
Rockville, MD 20850, or by email to [email protected].
Richard Kronick, Ph.D. David Meyers, M.D.
Director Acting Director
Agency for Healthcare Research and Quality Center for Evidence and Practice Improvement
Agency for Healthcare Research and Quality
Stephanie Chang, M.D., M.P.H.
Director and Task Order Officer
Evidence-based Practice Program
Center for Evidence and Practice Improvement
Agency for Healthcare Research and Quality
iv
Key Informants
In designing the study questions, the EPC consulted several Key Informants who represent
the end-users of research. The EPC sought the Key Informant input on the priority areas for
research and synthesis. Key Informants are not involved in the analysis of the evidence or the
writing of the report. Therefore, in the end, study questions, design, methodological approaches,
and/or conclusions do not necessarily represent the views of individual Key Informants.
Key Informants must disclose any financial conflicts of interest greater than $10,000 and any
other relevant business or professional conflicts of interest. Because of their role as end-users,
individuals with potential conflicts may be retained. The TOO and the EPC work to balance,
manage, or mitigate any conflicts of interest.
The list of Key Informants who participated in developing this report follows:
Nadera Ahmadzai, M.D., M.P.H., M.Sc. David Hopkins, M.D., M.P.H.
Senior Research Associate Medical Officer
OHRI Knowledge Synthesis Group Community Guide Branch
Centers for Disease Control and Prevention
Samantha Barker, B.S.
Senior Project Manager, Evidence Synthesis Susan Norris, M.D., M.Sc., M.P.H.
and Nutritionist Guidelines Review Committee Secretariat
Research Funding Program World Health Organization
Institute for Safety, Compensation and
Recovery Research Julie Obbagy, Ph.D., R.D.
Center for Nutrition Policy and Promotion
Lorne Becker, M.D. Nutrition Guidance and Analysis Division,
Emeritus Professor USDA
Department of Family Medicine
SUNY Upstate Medical University Julie Polisena, M.Sc.
Scientific Advisor
Kathryn M. Curtis, Ph.D. Canadian Agency for Drugs and
Health Scientist Technologies in Health
Division of Reproductive Health
Centers for Disease Control and Prevention Joanne Spahn, M.S., R.D., F.A.D.A.
Director
Chantelle Garritty, B.A., D.C.S., M.Sc. USDA Nutrition Evidence Analysis Library
Senior Operations Manager Division
Ottawa Methods Centre, Ottawa Hospital
Lesley Stewart, Ph.D.
Director
University of York Centre for Reviews and
Dissemination
v
Peer Reviewers
Prior to publication of the final evidence report, EPCs sought input from independent Peer
Reviewers without financial conflicts of interest. However, the conclusions and synthesis of the
scientific literature presented in this report does not necessarily represent the views of individual
reviewers.
Peer Reviewers must disclose any financial conflicts of interest greater than $10,000 and any
other relevant business or professional conflicts of interest. Because of their unique clinical or
content expertise, individuals with potential non-financial conflicts may be retained. The TOO
and the EPC work to balance, manage, or mitigate any potential non-financial conflicts of
interest identified.
The list of Peer Reviewers follows:
Nadera Ahmadzai, M.D., M.P.H., M.Sc. Tianjing Li, M.D., Ph.D., M.H.S.
Senior Research Associate, OHRI Assistant Professor
Knowledge Synthesis Group Center for Clinical Trials
Eric Bass, MD Department of Epidemiology
Director, Johns Hopkins Evidence-based Johns Hopkins Bloomberg School of Public
Practice Center Health
Professor of Medicine
Marian McDonagh, Pharm. D.
Kathryn M. Curtis, Ph.D. Associate Director, Pacific Northwest
Health Scientist, Division of Reproductive Evidence-based Practice Center
Health Professor of Medical Informatics & Clinical
Centers for Disease Control and Prevention Epidemiology, Oregon Health & Science
University
Paul Glasziou, MRCGP, FRACGP, Ph.D.,
M.B.B.S. Melissa McPheeters, Ph.D., M.P.H.
Director (CREBP), Faculty of Health Director, Vanderbilt Evidence-based
Sciences and Medicine Practice Center
Bond University Director, Epidemiology Track, M.P.H.
Research Associate Professor
David Hopkins, M.D., M.P.H. Department of Health Policy
Medical Officer, Community Guide Branch Vanderbilt University Medical Center
Centers for Disease Control and Prevention
Dawid Pieper, M.P.H.
Bob Kane, M.D. Universitaet Witten/Herdecke
Co-Director, Minnesota Evidence-based Institute for Research in Operative Medicine
Practice Center
School of Public Health Julie Polisena, M.Sc.
University of Minnesota Scientific Advisor, Canadian Agency for
Drugs and Technologies in Health
Public Comment
American Physical Therapy Association
vi
Paul Shekelle, Ph.D., M.D., M.P.H. Tom Trikalinos, M.D.
Director, Southern California Evidence- Director, Brown Evidence-based Practice
based Practice Center, RAND Corporation Center
Director, Center for Evidence-based
Joanne Spahn, M.S., R.D., FADA Medicine
Director, USDA Nutrition Evidence Associate Professor, Health Services, Policy
Analysis Library Division & Practice
Brown University
Jon Treadwell, Ph.D.
Associate Director, ECRI Institute Meera Viswanathan, Ph.D.
Evidence-based Practice Center Director, RTI-UNC Evidence-based Practice
Center
RTI International

vii
Integrating Bodies of Evidence: Existing Systematic
Reviews and Primary Studies
Introduction
In 2008, recognizing the exponential growth in the number of systematic reviews being
published, the need to update existing reviews, and the increasing time and money constraints, a
group of researchers across the Agency for Healthcare Research and Quality (AHRQ) Evidence-
based Practice Centers (EPC) program developed preliminary guidance on the appropriate role of
existing reviews in developing new reviews on related topics.1 This work identified a series of
issues to consider and developed guidance to address some of these issues, which subsequently
became codified as a chapter in the AHRQ EPC Program Methods Guide.2
In 2012, an EPC Methods Workgroup sought to identify remaining challenges in integrating
existing systematic reviews into new reviews. The Workgroup conducted targeted scans of
published and grey literature to identify guidance used or offered by organizations dedicated to
conducting systematic reviews or improving the systematic review process.3,4 Discussions with
EPC directors and staff were conducted to identify ways EPCs have found and used existing
systematic reviews. The Workgroup identified eight areas where additional guidance is needed
for reviewers considering integrating existing systematic reviews in new reviews:
• Criteria to identify when a new EPC review will add value to a field with many existing
reviews;
• Organizing principles for integrating primary (study-level) and secondary (systematic
review-level) evidence, (including templates for evidence tables);
• Guidelines for transparently reporting the methods used to identify, select, and decide
how best to utilize existing systematic reviews;
• Methods for minimizing bias in selecting prior reviews to use or integrate when there are
multiple existing reviews;
• Methods for minimizing bias in incorporating selected portions of an existing review;
• Qualitative and quantitative methods for summarizing bodies of evidence that include a
systematic review as the only or as one source of evidence;
• Robust quality assessments for existing systematic reviews (beyond AMSTAR5), and;
• Methods to grade the strength of evidence for bodies of evidence that include a
systematic review as the only or as one source of evidence.
We convened a new Workgroup in 2013 to develop recommendations on several of these
areas that the EPCs had identified as most pressing. Based on these identified needs and
preliminary discussions, our objective was to develop guidance on integrating bodies of evidence
from systematic reviews and primary studies in new reviews. Different uses of existing
systematic review, such as checking references and summarizing existing evidence in
introduction or discussion, have been adequately described in prior work.3,4 We specifically
focused on methods for using reviews when there are multiple reviews, assessing risk of bias of
primary studies in existing reviews, and summarizing and assessing the strength of bodies of
evidence that include or are limited to existing systematic reviews.3,4 The immediate intended
audience is the EPC program, but we hope that this guidance may be useful to all systematic
reviewers facing these issues.
1
Methods
Approach
We assembled a workgroup of EPC methodologists to develop recommendations on the
integration of existing systematic reviews in new reviews, building on the work of the previous
EPC Methods Workgroup. We sought information from an updated scan of the literature and
interviews with leaders in the field to inform consensus recommendations developed through
twice monthly conference calls.
Literature Search
The Scientific Resource Center (SRC) provides support for the AHRQ EPC Program for the
advancement of scientific methods, strategic planning, peer review, topic nomination and
education. As part of this work, the SRC curates a bibliographic database of nearly 10,000
citations on the methodology of systematic reviews and comparative effectiveness research,
dating back to the 1950s.6 We searched this database (22 April 2014) for publications that
included any of the following terms in the title, abstract, or descriptor: Overview; Umbrella;
Review of review; Use of secondary studies; Discordant review; Incorporating review; Multiple
systematic review; Review of systematic review; Relevant review; Synthesis of systematic
review; Secondary evidence; Synopsis of systematic; Synopsis of review. Citations were
screened first by the SRC informationist to remove material clearly not relevant and then by at
least one member of the Workgroup. We sought documents that provided guidance on the
integration of existing systematic reviews in systematic reviews. We were seeking literature that
could inform discussions and thus did not apply strict eligibility criteria. However, we did not
collect examples of how existing systematic reviews have been used; rather we used Key
Informant interviews to understand how reviews have been used. We also did not consider
related but separate topics such as methods for updating reviews or the conduct of reviews of
reviews (overviews).
Key Informant Interviews
We invited systematic reviewers, representatives from organizations that produce systematic
reviews and methodologists to participate in 60-minute telephone interviews. Workgroup
members interviewed 11 of these “key informants” (KIs) or thought leaders from organizations
that conduct or use systematic reviews. Each KI completed a conflict of interest disclosure form
prior to participation. Prior to initiating the interviews, we developed and piloted an interview
guide to focus the interviews (Appendix A), which includes a brief introduction of the
background of the workgroup, the purpose of the interview, and interview questions. The
interview guide was sent to KIs prior to the call. The interview questions covered three general
topics:
• Using multiple existing reviews
• Assessing risk of bias
• Summarizing and assessing bodies of evidence.

We developed five scenarios or case studies outlining alternative actions depicting a range of
approaches to integrating an existing review into a new review to help frame the discussions.
2
These scenarios, which assume that at least one relevant existing review has been identified that
is considered of acceptable quality, are not mutually exclusive:
• Scenario 1: Use review without modifying or adding new studies
• Scenario 2: Use review and add new studies
• Scenario 3: Use review with new or modified analysis
• Scenario 4: Use selected elements of review
• Scenario 5: Do not use review

Each interview was recorded and transcribed. One investigator analyzed transcripts of the
interviews for key themes using NVivo10 software (QSR International) and these themes were
reviewed by at least one other investigator. In the Results section, we present the themes by the
three general topics described above. More detail about the issues focused on within each of the
themes is provided in Appendix B, organized by both general topics and scenarios.
Development of Recommendations
All workgroup members reviewed the themes and examples from the KI interviews.
Conference calls were held twice a month. We developed recommendations in an iterative
manner until consensus was reached. As with all guidance for the conduct of systematic reviews,
we found very little evidence on which to base our recommendations; therefore, we focused on
providing guidance for areas in which the workgroup came to consensus on minimum standards.
For areas in which there was less certainty that following the guidance would be worth the effort
required, the workgroup provided suggested actions that review authors may elect not to follow
due to resource or other constraints.
Results
Literature Search
After screening the 470 citations found in the methods research database, we identified no
literature relevant to informing our discussions, other than the previous EPC methods work.1-4
Synthesis of Key Informant Interviews
We interviewed 11 KIs from various organizations that conduct systematic review. While
one organization noted that it chooses not to include any existing systematic reviews in its
reviews, most organizations described a process to evaluate and include existing reviews, though
none of them has published guidance on this issue. One organization mentioned using the prior
EPC methods work in this area.1 Key themes from the interviews are organized by the three
general topics in this section and more detail is presented in Appendix B.
Using Multiple Existing Reviews
KIs reported that it is common to identify multiple relevant existing systematic reviews and
that they would typically use the “best” review rather than include all existing reviews. KIs cited
several considerations in deciding which was the “best” among a group of reviews. In general,
they noted that priority is given to reviews that most closely match the current review; scope
(populations, interventions, and outcomes of interest), inclusion/exclusion criteria, and methods.
3