Pettigrew-Roberts-SR-in-the-Soc-Sc

354 Pages·2021·1.78 MB·English

Systematic Reviews
in the Social Sciences

A PRACTICAL GUIDE

Mark Petticrew and Helen Roberts

Petticrew / Assessing Adults with Intellecutal Disabilities
1405121106_01_pre-toc
Final Proof
page 3
4.8.2005
1:07am

Systematic Reviews in the Social Sciences

Petticrew / Assessing Adults with Intellecutal Disabilities
1405121106_01_pre-toc
Final Proof
page 1
4.8.2005
1:07am

Petticrew / Assessing Adults with Intellecutal Disabilities
1405121106_01_pre-toc
Final Proof
page 2
4.8.2005
1:07am

Systematic Reviews
in the Social Sciences

A PRACTICAL GUIDE

Mark Petticrew and Helen Roberts

Petticrew / Assessing Adults with Intellecutal Disabilities
1405121106_01_pre-toc
Final Proof
page 3
4.8.2005
1:07am

� 2006 by Mark Petticrew and Helen Roberts

BLACKWELL PUBLISHING
350 Main Street, Malden, MA 02148-5020, USA
9600 Garsington Road, Oxford OX4 2DQ, UK
550 Swanston Street, Carlton, Victoria 3053, Australia

The right of Mark Petticrew and Helen Roberts to be identified as the Authors of this Work
has been asserted in accordance with the UK Copyright, Designs, and Patents Act 1988.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval
system, or transmitted, in any form or by any means, electronic, mechanical, photo-
copying, recording or otherwise, except as permitted by the UK Copyright, Designs,
and Patents Act 1988, without the prior permission of the publisher.

First published 2006 by Blackwell Publishing Ltd

1
2006

Library of Congress Cataloging-in-Publication Data

Petticrew, Mark.
Systematic reviews in the social sciences : a practical guide/Mark Petticrew and Helen
Roberts.
p. cm.
Includes bibliographical references and index.
ISBN-13: 978-1-4051-2110-1 (hard cover: alk. paper)
ISBN-10: 1-4051-2110-6 (hard cover: alk. paper)
ISBN-13: 978-1-4051-2111-8 (pbk. : alk. paper)
ISBN-10: 1-4051-2111-4 (pbk. : alk. paper)
1. Social sciences—Research—Methodology. 2. Social sciences—Statistical methods.
I. Roberts, Helen, 1949–II. Title.

H62.P457 2005
300’.72’3—dc22
2005011632
A catalogue record for this title is available from the British Library.

Set in 10.5pt/12.5pt Bembo
by SPI Publisher Services, Pondicherry, India
Printed and bound in the United Kingdom
by TJ International, Padstow, Cornwall

The publisher’s policy is to use permanent paper from mills that operate a sustainable
forestry policy, and which has been manufactured from pulp processed using acid-free
and elementary chlorine-free practices. Furthermore, the publisher ensures that the text
paper and cover board used have met acceptable environmental accreditation standards.

For further information on
Blackwell Publishing, visit our website:
www.blackwellpublishing.com

Petticrew / Assessing Adults with Intellecutal Disabilities
1405121106_01_pre-toc
Final Proof
page 4
4.8.2005
1:07am

Contents

Foreword – William R. Shadish
vi
Acknowledgments
x
Preface
xiii

Chapter 1
Why do we need systematic reviews?
1
Chapter 2
Starting the review: Refining the question
and defining the boundaries
27
Chapter 3
What sorts of studies do I include in the review?
Deciding on the review’s inclusion/exclusion criteria
57
Chapter 4
How to find the studies: The literature search
79
Chapter 5
How to appraise the studies: An introduction to
assessing study quality
125
Chapter 6
Synthesizing the evidence
164
Chapter 7
Exploring heterogeneity and publication bias
215
Chapter 8
Disseminating the review
247
Chapter 9
Systematic reviews: Urban myths and fairy tales
265
Glossary
277
Appendix 1 The review process (and some questions to ask
before starting a review)
284
Appendix 2 MOOSE Guidelines
288
Appendix 3 Example of a flow diagram from a systematic review
291
Appendix 4 Example data extraction form
293
Appendix 5 Variations in the quality of systematic reviews
296

Bibliography
298
Index
324

Petticrew / Assessing Adults with Intellecutal Disabilities
1405121106_02_toc
Final Proof
page 5
4.8.2005
1:07am

Foreword

The idea of evidence-based practice is not new. For example, nearly 60
years ago, the scientist-practitioner model of clinical psychology in the US
was based in part on the idea that clinical psychologists would scientifically
investigate the effectiveness of their own treatments, and use what proved to
be effective. Around 100 years ago, the British Army asked Sir Karl Pearson
to review the evidence about the effects of a new typhoid vaccine, to help
them decide whether to adopt that vaccine more widely.1 Such relatively
isolated examples are common for hundreds of years. What is new is the
widespread acceptance that evidence-based practice is practice whose time
has come. Whether it be through the Cochrane Collaboration, the Camp-
bell Collaboration, the What Works Clearinghouse in the US Department
of Education, or any of a host of similar efforts, the idea that practices and
policies that are supported by scientific evidence can be identified and
disseminated has now been institutionalized. That in itself is a remarkable
accomplishment.
In this context, this book by Petticrew and Roberts is a timely gem, full
of detailed history, pithy quotes, marvelous examples, compelling argu-
ments, practical advice, and transparent demonstrations of how to do one
of the most common ways of putting the evidence into evidence-based
practice – the systematic review. Practical advice leaps out at the reader from
every chapter, with novel information I have not seen in related works.
Let me start with reference to the key justification for systematic reviews,
that ‘‘single studies taken in isolation are often seriously misleading’’
(chapter 1, this volume). Or as I tend to put it: Never trust the results of
single studies. One need only graph a frequency distribution of effect sizes to
see that some studies will show unusually positive effects, and some un-
usually negative effects, substantially by virtue of chance. Yet the public, the
media, and even other scientists routinely place great faith in single studies.

Petticrew / Assessing Adults with Intellecutal Disabilities
1405121106_03_posttoc
Final Proof
page 6
4.8.2005
1:06am

As a result, ineffective or inaccurate policies, practices, or ideas are adopted,
accurate or effective ones are dismissed, and all parties get confused. The
analogy that Petticrew and Roberts make with a survey is a good one. We
would never take seriously the results of a sample survey where the sample
was of a single person, and we all understand why – we will often get a
different answer by asking a different person. The same is true in systematic
reviews. We will often get a different answer when we look at a different
study. In sample surveys, only a large sample is capable of giving us a reliable
answer. In reviewing literature on a question, only the systematic review
process is capable of helping to clarify where the answer really lies.
Yet as Petticrew and Roberts point out in chapter 2 and elaborate in
chapter 3, we have had an unnecessarily narrow view of what a systematic
review can do – answer questions about cause and effect relationships.
Systematic reviews need not only address what works, as important as that
question is. Such reviews can address the evidence about nearly any kind
of question, for example, about the prevalence of a problem, the correlation
of two variables to each other, the implementation of a program, the
meaning program stakeholders ascribe to a program or its context, the causal
mediating processes that help us understand why it works, and program
costs, to name just a few. Similarly, our understanding of the question of
what works has often been too narrow. That question involves high quality
evidence about cause and effect relationships, to be sure. But we are also
informed about what works by, for example, knowledge that some stake-
holder groups refuse to participate, that costs are too high to be practical, or
that the need for the treatment is questionable.
What is less clear is whether we have well-developed technologies for
addressing some of these questions. The technological problem is twofold,
how to integrate studies that are all of one kind (e.g., about need) with each
other, and how to integrates studies of one kind (e.g., need) with studies of
another kind (e.g., treatment effectiveness). The former problem is relat-
ively unproblematic for most kinds of quantitative studies, but is far less clear
for qualitative studies. The latter problem is almost completely unaddressed.
It is this context that makes me so pleased to see the effort Petticrew and
Roberts have put into both these problems. Many people talk about the
value of integrating quantitative and qualitative studies, for example, but
actual examples of such integration are few and far between. Petticrew and
Roberts lay out the logic of such integrations, some of the methods that
might be used for doing them, along with examples. And they don’t shy
away from the difficulties such as a lack of standardized search terms to use
in locating qualitative studies on a given topic. For many years, when critics
have decried the narrow focus on experiments in systematic reviews of what

FOREWORD
vii

Petticrew / Assessing Adults with Intellecutal Disabilities
1405121106_03_posttoc
Final Proof
page 7
4.8.2005
1:06am

works, I have responded by saying we are working on that problem. Now
I can actually point to the work.
I was also pleased to see the huge number of references in chapter 5 about
assessing the quality of different kinds of research designs. Just the compila-
tion itself is a major contribution. Yet we should remember that quality can
mean two different things: (a) quality of the implementation of the method
(e.g., did the primary study author implement the qualitative method
properly); and (b) quality of the inferences from the method (do the
methods used support a causal inference). My sense is that the latter kinds
of quality scales are pretty common for experiments, though serious ques-
tion remains about whether those scales are well enough developed to
warrant widespread use. But inference quality scales are far less well devel-
oped for qualitative methods, where the available scales seem more con-
cerned with whether the methods were implemented well. The problem is
that good implementation of a method is orthogonal to whether that
method supports a given inference. The best-implemented randomized
experiment is rarely going to tell us much about the need for an interven-
tion, just as the best implemented ethnography is rarely going to yield a
strong inference about cause-and-effect relationships.
Finally, chapter 8 on dissemination and use of systematic reviews raises a
crucial issue. Too often we assume that policymakers and practitioners are
likely to adopt interventions that are shown to work. That assumption is at
best only partly right. We learned that the hard way in the field of program
evaluation in the 1960s and 1970s. The early emphasis in that field was on
experimentation to determine effective interventions that could be adopted
as policy. We quickly found out that they were frequently not adopted even
when they were effective, and began to study the process of use itself.2 The
fate of Fairweather’s Lodge for the chronically mentally ill is illustrative.3

That intervention was far more cost effective than the alternatives, was well-
liked by patients and successful on most outcome criteria; but it was
relatively little used despite extensive dissemination funding because it had
no funding mechanism, conflicted with the extant treatment institutions,
and required citizens to accept a house of patients next door. Adopting
effective practices involves far more than knowing what works, and Petti-
crew and Roberts do a commendable job of laying out the issues. The
lessons learned are remarkably similar to those we learned in program
evaluation years ago, and we would do well to revisit the modern versions
of that literature (e.g., Patton)4 to help us understand more about how to
make our work useful.
As you can see, I am a fan of what Petticrew and Roberts have done in
this book. They lay out a broad and ambitious agenda for the art and science

viii
FOREWORD

Petticrew / Assessing Adults with Intellecutal Disabilities
1405121106_03_posttoc
Final Proof
page 8
4.8.2005
1:06am

of systematic reviews, an agenda that cogently presages issues that I think
will be at the cutting edge of this field of inquiry in the coming decades.
I hope you enjoy it as much as I did.
William R. Shadish
University of California, Merced

REFERENCES

1. Pearson, K. Report on certain enteric fever inoculation statistics. British Medical
Journal 1904, 3: 1243–6.
2. Shadish, W. R., Cook, T. D., and Leviton, L. C. Foundations of program evalu-
ation: Theories of practice. Newbury Park, CA: Sage Publications, 1991.
3. Shadish, W. R. Policy research: Lessons from the implementation of deinstitu-
tionalization. American Psychologist 1984, 39: 725–38.
4. Patton, M. Q. Utilization-focused evaluation (3rd edn.). Newbury Park, CA: Sage
Publications, 1996.

FOREWORD
ix

Petticrew / Assessing Adults with Intellecutal Disabilities
1405121106_03_posttoc
Final Proof
page 9
4.8.2005
1:06am

Acknowledgments

Writing books incurs a large number of debts. This one is no different, and
we would particularly like to thank Susan Kennedy and Rodney Barker,
who have gone well beyond the usual demands of being our partners by
reading, commenting, and constructively criticizing.
Professor Will Shadish, one of the foremost meta-analysts in the USA
readily and generously agreed to write a foreword. The anonymous peer
reviewers of our proposal and, in particular, the reviewer of our final
manuscript contributed a great deal to our thinking and the final product.
At our request, Blackwell approached them to ask if they would forgo
anonymity so that we could acknowledge them by name, and we are
happy to say that they agreed to this. Many thanks then, to Trevor Sheldon,
at the University of York, who nobly reviewed both the proposal and the
book; and to the proposal reviewers Jim Connelly, University of Leeds,
Emma Davies, Auckland University of Technology, David DuBois, Uni-
versity of Illinois at Chicago, Loraine Gelsthorpe, University of Cambridge,
David Gough, Institute of Education, University of London, Geraldine
Macdonald, Commission for Social Care Inspection, Susan Michie, Uni-
versity College London, Deborah Phillips, Georgetown University, and
Maggie Savin-Baden, Coventry University. We are also grateful to Kate
Pool from the Society of Authors.
Our colleagues in Glasgow and City Universities, our co-researchers and
colleagues on the ESRC-funded project on narrative synthesis: Jennie
Popay, Amanda Sowden, Nicky Britten, and the late Sally Baldwin, Lisa
Arai, Mark Rodgers, and Katrina Roen, and our colleagues in the ESRC
Evidence Network, and the Cochrane and Campbell Collaborations (par-
ticularly those whose work we have quoted) have all been parties to
discussions and conversations that have helped inform our work.

Petticrew / Assessing Adults with Intellecutal Disabilities
1405121106_03_posttoc
Final Proof
page 10
4.8.2005
1:06am

Mark Petticrew would particularly like to thank colleagues in the ESRC
Centre for Evidence-Based Public Health Policy, and in his research pro-
gram (funded by the Chief Scientist Office of the Scottish Executive
Department of Health), whose work and thinking the book draws on:
among them Pam Attree, Beth Milton, Clare Bambra, Frances Drever,
Matt Egan, Hilary Graham, Val Hamilton, Sally Macintyre, Liz McDer-
mott, David Morrison, David Ogilvie, Sian Thomas, Hilary Thomson, and
Margaret Whitehead. Colleagues elsewhere who read and commented on
parts of the book, or who kindly gave permission to use their work, include
Iain Chalmers, Iain Crombie, the EPOC (Effective Practice and Organiza-
tion of Care) group of the Cochrane Collaboration, Simon Gilbody, David
Gough and colleagues at the EPPI Centre, Rebecca Ivers and colleagues,
Betsy Kristjansson, Geoffrey Nelson and colleagues, Anthony Petrosino,
Steve Platt, Scott Reeves, Roberta Scherer, Fujian Song, Peter Tugwell,
and Paul Wilson. Thanks for permission to reproduce various figures or
tables go to John Wiley & Sons Ltd (the Forest plots in chapters 6 and 7,
which are copyright of the Cochrane Library), The Sainsbury Archive
(Figure 6.2 in chapter 6), Hawaii Medical Library, the National Portrait
Gallery (for the portrait of Francis Bacon in chapter 4), the British Medical
Journal (for permission to reproduce material in chapter 9), David Ogilvie
(the photograph in chapter 6), and the Centre for Reviews and Dissemin-
ation (CRD) at the University of York for permission to reproduce material
from their systematic reviews guidance (CRD Report 4). Thanks also to
Mary Robins for library support.
Helen Roberts is particularly grateful to colleagues at City University,
especially current and past colleagues in the Child Health Research and
Policy Unit, and to fellow members of the What Works for Children
‘‘node’’ funded as part of the ESRC evidence network – Trevor Sheldon
at the University of York, Kristin Liabo, Madeleine Stevens, Greg Khine,
and Alison Moore at City University and Di McNeish, Tony Newman,
Sarah Frost, Angela Hutton, and Philippa Roberts at Barnardo’s. Shruti
Uppal provided valuable assistance with reference and Web checking.
Helen has learned a great deal from colleagues at the EPPI Centre at the
University of London Institute of Education and others who contributed to
the seminar series on research evidence organized by the Evidence Net-
work, funded by the UK Health Development Agency. Other colleagues
with whom Helen has worked on systematic reviews including Carol
Joughin, Gabrielle Laing, Stuart Logan, Sandra Dowling, Catherine Law,
Jos Kleijnen, Patricia Lucas, Janis Baird, and David Fisher have helped her
thinking while working on joint systematic reviews. Ruth Gilbert, Ian
Roberts, and Dennis Cheek all responded quickly and helpfully to requests

ACKNOWLEDGMENTS
xi

Petticrew / Assessing Adults with Intellecutal Disabilities
1405121106_03_posttoc
Final Proof
page 11
4.8.2005
1:06am

for advice. Helen would also like to thank her former colleagues in the
R&D team in Barnardo’s with whom she worked on some of the material
on which we draw in chapter 8, and to ESRC and David Walker for
allowing us to draw on their communications toolkit and related work.
Thanks to Getpetsonline for permission to use the photograph in figure
2.2 and to the Thomas A. Edison papers archive at Rutgers University for
permission to use the image in Box 4.5.
Sarah Bird, Will Maddox, Joanna Pyke, Rhonda Pearce, Mervyn Tho-
mas, Helen Kemp, and Zeb Korycinska from Blackwell looked after this
book as it moved through publication. We thank them for their careful
work.
Mark Petticrew
Helen Roberts

xii
ACKNOWLEDGMENTS

Petticrew / Assessing Adults with Intellecutal Disabilities
1405121106_03_posttoc
Final Proof
page 12
4.8.2005
1:06am

Preface

Research studies use a variety of methods, are of variable quality and may
appear to have contradictory findings . . . Research information can seem to
users like small jigsaw pieces in a box where there may be several pictures,
several duplicates and several missing pieces. Under these circumstances it can
be difficult for the researcher, research funder or user to make sense of the
research or take stock of the knowledge base. (Sheldon, 1998)1

Literature reviews have many purposes. They can examine old theories and
propose new ones, consider where the balance of evidence lies in relation to
a particular topic, and provide a basis for recommendations for interven-
tions, (such as teaching in smaller or larger classes, running an anti-bullying
program, or offering parenting education). They can provide guidance to
researchers planning future studies, and provide convenient summaries of
the literature on a particular issue. They can also be used to examine
methodically the reasons why different studies addressing the same question
sometimes reach different conclusions – a common and frustrating problem
for both researchers and research users.
This book is a guide to planning and conducting a particular type of
literature review, one that is increasingly used as a scientific tool: the
systematic literature review. The book is aimed at social science researchers,
but it provides a more general discussion of systematic reviews for those
who want to use and understand them, but don’t necessarily want to do one
themselves. Most of the recent interest in systematic reviews focuses on
reviews of the effectiveness of interventions, reflected in the growth of
initiatives like the Campbell and Cochrane Collaborations. This book
therefore focuses on reviews of effectiveness, but not exclusively. Systematic
reviews are also widely used to synthesize other sorts of evidence, for
example in order to answer questions about etiology (causes of problems),

Petticrew / Assessing Adults with Intellecutal Disabilities
1405121106_03_posttoc
Final Proof
page 13
4.8.2005
1:06am