Systematic Reviews in the Social Sciences: A Practical Guide

350 Pages·2005·2.371 MB·English

by  Mark Petticrew, Helen Roberts(auth.)

Systematic Reviews in the Social Sciences
Systematic Reviews
in the Social Sciences
A PRACTICAL GUIDE
Mark Petticrew and Helen Roberts
(cid:2)2006byMarkPetticrewandHelenRoberts
BLACKWELLPUBLISHING
350MainStreet,Malden,MA02148-5020,USA
9600GarsingtonRoad,OxfordOX42DQ,UK
550SwanstonStreet,Carlton,Victoria3053,Australia
TherightofMarkPetticrewandHelenRobertstobeidentifiedastheAuthorsofthisWork
hasbeenassertedinaccordancewiththeUKCopyright,Designs,andPatentsAct1988.
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrieval
system, or transmitted, in any form or by any means, electronic, mechanical, photo-
copying, recording or otherwise, except as permitted by the UK Copyright, Designs,
andPatentsAct1988,withoutthepriorpermissionofthepublisher.
Firstpublished2006byBlackwellPublishingLtd
1 2006
LibraryofCongressCataloging-in-PublicationData
Petticrew,Mark.
Systematicreviewsinthesocialsciences:apracticalguide/MarkPetticrewandHelen
Roberts.
p.cm.
Includesbibliographicalreferencesandindex.
ISBN-13:978-1-4051-2110-1(hardcover:alk.paper)
ISBN-10:1-4051-2110-6(hardcover:alk.paper)
ISBN-13:978-1-4051-2111-8(pbk.:alk.paper)
ISBN-10:1-4051-2111-4(pbk.:alk.paper)
1. Social sciences—Research—Methodology. 2. Social sciences—Statistical methods.
I.Roberts,Helen,1949–II.Title.
H62.P4572005
300’.72’3—dc22
2005011632
AcataloguerecordforthistitleisavailablefromtheBritishLibrary.
Setin10.5pt/12.5ptBembo
bySPIPublisherServices,Pondicherry,India
PrintedandboundintheUnitedKingdom
byTJInternational,Padstow,Cornwall
The publisher’s policy is to use permanent paper from mills that operate a sustainable
forestry policy, and which has been manufactured from pulp processed using acid-free
andelementarychlorine-freepractices.Furthermore,thepublisherensuresthatthetext
paperandcoverboardusedhavemetacceptableenvironmentalaccreditationstandards.
Forfurtherinformationon
BlackwellPublishing,visitourwebsite:
www.blackwellpublishing.com
Contents
Foreword – William R. Shadish vi
Acknowledgments x
Preface xiii
Chapter 1 Why do we need systematic reviews? 1
Chapter 2 Starting the review: Refining the question
and defining the boundaries 27
Chapter 3 What sorts of studies do I include in the review?
Deciding on the review’s inclusion/exclusion criteria 57
Chapter 4 How to find the studies: The literature search 79
Chapter 5 How to appraise the studies: An introduction to
assessing study quality 125
Chapter 6 Synthesizing the evidence 164
Chapter 7 Exploring heterogeneity and publication bias 215
Chapter 8 Disseminating the review 247
Chapter 9 Systematic reviews: Urban myths and fairy tales 265
Glossary 277
Appendix 1 The review process (and some questions to ask
before starting a review) 284
Appendix 2 MOOSE Guidelines 288
Appendix 3 Example of a flow diagram from a systematic review 291
Appendix 4 Example data extraction form 293
Appendix 5 Variations in the quality of systematic reviews 296
Bibliography 298
Index 324
Foreword
The idea of evidence-based practice is not new. For example, nearly 60
years ago, the scientist-practitioner model of clinical psychology in the US
was based in part on the idea that clinical psychologists would scientifically
investigatetheeffectivenessoftheirowntreatments,andusewhatprovedto
beeffective.Around100yearsago,theBritishArmyaskedSirKarlPearson
to review the evidence about the effects of a new typhoid vaccine, to help
them decide whether to adopt that vaccine more widely.1 Such relatively
isolated examples are common for hundreds of years. What is new is the
widespread acceptance that evidence-based practice is practice whose time
has come. Whether it be through the Cochrane Collaboration, the Camp-
bell Collaboration, the What Works Clearinghouse in the US Department
of Education, or any of a host of similar efforts, the idea that practices and
policies that are supported by scientific evidence can be identified and
disseminated has now been institutionalized. That in itself is a remarkable
accomplishment.
In this context, this book by Petticrew and Roberts is a timely gem, full
of detailed history, pithy quotes, marvelous examples, compelling argu-
ments, practical advice, and transparent demonstrations of how to do one
of the most common ways of putting the evidence into evidence-based
practice–thesystematicreview.Practicaladviceleapsoutatthereaderfrom
every chapter, with novel information I have not seen in related works.
Letmestartwithreferencetothekeyjustificationforsystematicreviews,
that ‘‘single studies taken in isolation are often seriously misleading’’
(chapter 1, this volume). Or as I tend to put it: Never trust the results of
singlestudies.Oneneedonlygraphafrequencydistributionofeffectsizesto
see that some studies will show unusually positive effects, and some un-
usuallynegativeeffects,substantiallybyvirtueofchance.Yetthepublic,the
media,andeven other scientists routinelyplace greatfaith in single studies.
FOREWORD vii
Asaresult,ineffectiveorinaccuratepolicies,practices,orideasareadopted,
accurate or effective ones are dismissed, and all parties get confused. The
analogy that Petticrew and Roberts make with a survey is a good one. We
would never take seriously the results of a sample survey where the sample
was of a single person, and we all understand why – we will often get a
differentanswerbyaskingadifferentperson.Thesameistrueinsystematic
reviews. We will often get a different answer when we look at a different
study.Insamplesurveys,onlyalargesampleiscapableofgivingusareliable
answer. In reviewing literature on a question, only the systematic review
process is capable of helping to clarify where the answer really lies.
Yet as Petticrew and Roberts point out in chapter 2 and elaborate in
chapter 3, we have had an unnecessarily narrow view of what a systematic
review can do – answer questions about cause and effect relationships.
Systematic reviews need not only address what works, as important as that
question is. Such reviews can address the evidence about nearly any kind
ofquestion,forexample,abouttheprevalenceofaproblem,thecorrelation
of two variables to each other, the implementation of a program, the
meaningprogramstakeholdersascribetoaprogramoritscontext,thecausal
mediating processes that help us understand why it works, and program
costs, to name just a few. Similarly, our understanding of the question of
whatworkshasoftenbeentoonarrow.Thatquestioninvolveshighquality
evidence about cause and effect relationships, to be sure. But we are also
informed about what works by, for example, knowledge that some stake-
holdergroupsrefusetoparticipate,thatcostsaretoohightobepractical,or
that the need for the treatment is questionable.
What is less clear is whether we have well-developed technologies for
addressing some of these questions. The technological problem is twofold,
howtointegratestudiesthatareallofonekind(e.g.,aboutneed)witheach
other,andhowtointegratesstudiesofonekind(e.g.,need)withstudiesof
another kind (e.g., treatment effectiveness). The former problem is relat-
ivelyunproblematicformostkindsofquantitativestudies,butisfarlessclear
forqualitativestudies.Thelatterproblemisalmostcompletelyunaddressed.
It is this context that makes me so pleased to see the effort Petticrew and
Roberts have put into both these problems. Many people talk about the
value of integrating quantitative and qualitative studies, for example, but
actualexamplesof such integration are few andfarbetween.Petticrew and
Roberts lay out the logic of such integrations, some of the methods that
might be used for doing them, along with examples. And they don’t shy
away from the difficulties such as a lack of standardized search terms to use
inlocatingqualitativestudiesonagiventopic.Formanyyears,whencritics
havedecriedthenarrowfocusonexperimentsinsystematicreviewsofwhat
viii FOREWORD
works, I have responded by saying we are working on that problem. Now
I can actually point to the work.
Iwasalsopleasedtoseethehugenumberofreferencesinchapter5about
assessingthequalityofdifferentkindsofresearchdesigns.Justthecompila-
tionitselfisamajorcontribution.Yetweshouldrememberthatqualitycan
meantwodifferentthings:(a)qualityoftheimplementationofthemethod
(e.g., did the primary study author implement the qualitative method
properly); and (b) quality of the inferences from the method (do the
methods used support a causal inference). My sense is that the latter kinds
of quality scales are pretty common for experiments, though serious ques-
tion remains about whether those scales are well enough developed to
warrant widespread use. But inference quality scales are far less well devel-
oped for qualitative methods, where the available scales seem more con-
cernedwithwhetherthemethodswereimplementedwell.Theproblemis
that good implementation of a method is orthogonal to whether that
method supports a given inference. The best-implemented randomized
experiment is rarely going to tell us much about the need for an interven-
tion, just as the best implemented ethnography is rarely going to yield a
strong inference about cause-and-effect relationships.
Finally, chapter 8 on dissemination and use of systematic reviews raises a
crucial issue. Too often we assume that policymakers and practitioners are
likelytoadoptinterventions thatareshown towork. Thatassumptionisat
bestonlypartlyright.Welearnedthatthehardwayinthefieldofprogram
evaluation in the 1960s and 1970s. The early emphasis in that field was on
experimentationtodetermineeffectiveinterventionsthatcouldbeadopted
aspolicy.Wequicklyfoundoutthattheywerefrequentlynotadoptedeven
whentheywereeffective,andbegantostudytheprocessofuseitself.2The
fate of Fairweather’s Lodge for the chronically mentally ill is illustrative.3
Thatinterventionwasfarmorecosteffectivethanthealternatives,waswell-
liked by patients and successful on most outcome criteria; but it was
relatively little used despite extensive dissemination funding because it had
no funding mechanism, conflicted with the extant treatment institutions,
and required citizens to accept a house of patients next door. Adopting
effective practices involves far more than knowing what works, and Petti-
crew and Roberts do a commendable job of laying out the issues. The
lessons learned are remarkably similar to those we learned in program
evaluation years ago, and we would do well to revisit the modern versions
of that literature (e.g., Patton)4 to help us understand more about how to
make our work useful.
As you can see, I am a fan of what Petticrew and Roberts have done in
thisbook.Theylayoutabroadandambitiousagendafortheartandscience
FOREWORD ix
of systematic reviews, an agenda that cogently presages issues that I think
will be at the cutting edge of this field of inquiry in the coming decades.
I hope you enjoy it as much as I did.
William R. Shadish
University of California, Merced
REFERENCES
1. Pearson,K.Reportoncertainentericfeverinoculationstatistics.BritishMedical
Journal1904,3:1243–6.
2. Shadish, W. R., Cook, T. D., and Leviton, L. C. Foundations of program evalu-
ation:Theoriesofpractice.NewburyPark,CA:SagePublications,1991.
3. Shadish, W. R. Policy research: Lessons from the implementation of deinstitu-
tionalization.AmericanPsychologist1984,39:725–38.
4. Patton,M.Q.Utilization-focusedevaluation(3rdedn.).NewburyPark,CA:Sage
Publications,1996.
Acknowledgments
Writingbooksincursalargenumberofdebts.Thisoneisnodifferent,and
we would particularly like to thank Susan Kennedy and Rodney Barker,
who have gone well beyond the usual demands of being our partners by
reading, commenting, and constructively criticizing.
Professor Will Shadish, one of the foremost meta-analysts in the USA
readily and generously agreed to write a foreword. The anonymous peer
reviewers of our proposal and, in particular, the reviewer of our final
manuscript contributed a great deal to our thinking and the final product.
At our request, Blackwell approached them to ask if they would forgo
anonymity so that we could acknowledge them by name, and we are
happytosaythattheyagreedtothis.Manythanksthen,toTrevorSheldon,
at the University of York, who nobly reviewed both the proposal and the
book; and to the proposal reviewers Jim Connelly, University of Leeds,
Emma Davies, Auckland University of Technology, David DuBois, Uni-
versityofIllinoisatChicago,LoraineGelsthorpe,UniversityofCambridge,
David Gough, Institute of Education, University of London, Geraldine
Macdonald, Commission for Social Care Inspection, Susan Michie, Uni-
versity College London, Deborah Phillips, Georgetown University, and
Maggie Savin-Baden, Coventry University. We are also grateful to Kate
Pool from the Society of Authors.
OurcolleaguesinGlasgowandCityUniversities,ourco-researchersand
colleagues on the ESRC-funded project on narrative synthesis: Jennie
Popay, Amanda Sowden, Nicky Britten, and the late Sally Baldwin, Lisa
Arai, Mark Rodgers, and Katrina Roen, and our colleagues in the ESRC
Evidence Network, and the Cochrane and Campbell Collaborations (par-
ticularly those whose work we have quoted) have all been parties to
discussions and conversations that have helped inform our work.
ACKNOWLEDGMENTS xi
MarkPetticrewwouldparticularlyliketothankcolleaguesintheESRC
Centre for Evidence-Based Public Health Policy, and in his research pro-
gram (funded by the Chief Scientist Office of the Scottish Executive
Department of Health), whose work and thinking the book draws on:
among them Pam Attree, Beth Milton, Clare Bambra, Frances Drever,
Matt Egan, Hilary Graham, Val Hamilton, Sally Macintyre, Liz McDer-
mott,DavidMorrison,DavidOgilvie,SianThomas,HilaryThomson,and
Margaret Whitehead. Colleagues elsewhere who read and commented on
partsofthebook,orwhokindlygavepermissiontousetheirwork,include
IainChalmers,IainCrombie,the EPOC(Effective Practice andOrganiza-
tionofCare)groupoftheCochraneCollaboration,SimonGilbody,David
Gough and colleagues at the EPPI Centre, Rebecca Ivers and colleagues,
Betsy Kristjansson, Geoffrey Nelson and colleagues, Anthony Petrosino,
Steve Platt, Scott Reeves, Roberta Scherer, Fujian Song, Peter Tugwell,
and Paul Wilson. Thanks for permission to reproduce various figures or
tables go to John Wiley & Sons Ltd (the Forest plots in chapters 6 and 7,
which are copyright of the Cochrane Library), The Sainsbury Archive
(Figure 6.2 in chapter 6), Hawaii Medical Library, the National Portrait
Gallery (for the portrait of Francis Bacon in chapter 4), the British Medical
Journal (for permission to reproduce material in chapter 9), David Ogilvie
(the photograph in chapter 6), and the Centre for Reviews and Dissemin-
ation(CRD)attheUniversityofYorkforpermissiontoreproducematerial
from their systematic reviews guidance (CRD Report 4). Thanks also to
Mary Robins for library support.
Helen Roberts is particularly grateful to colleagues at City University,
especially current and past colleagues in the Child Health Research and
Policy Unit, and to fellow members of the What Works for Children
‘‘node’’ funded as part of the ESRC evidence network – Trevor Sheldon
at the University of York, Kristin Liabo, Madeleine Stevens, Greg Khine,
and Alison Moore at City University and Di McNeish, Tony Newman,
Sarah Frost, Angela Hutton, and Philippa Roberts at Barnardo’s. Shruti
Uppal provided valuable assistance with reference and Web checking.
Helen has learned a great deal from colleagues at the EPPI Centre at the
UniversityofLondonInstituteofEducationandotherswhocontributedto
the seminar series on research evidence organized by the Evidence Net-
work, funded by the UK Health Development Agency. Other colleagues
with whom Helen has worked on systematic reviews including Carol
Joughin, Gabrielle Laing, Stuart Logan, Sandra Dowling, Catherine Law,
Jos Kleijnen, Patricia Lucas, Janis Baird, and David Fisher have helped her
thinking while working on joint systematic reviews. Ruth Gilbert, Ian
Roberts,andDennisCheekallrespondedquicklyandhelpfullytorequests