Understanding the new statistics : effect sizes, confidence intervals, and meta-analysis

537 Pages·2012·8.59 MB·English

by  Geoff Cumming

Understanding
The New Statistics
E S , c
ffEct izES onfidEncE
i , M -a
ntErvalS and Eta nalySiS

Geoff Cumming
La Trobe University, Melbourne, Australia
Multivariate Applications Series
Sponsored by the Society of Multivariate Experimental Psychology, the goal
of this series is to apply complex statistical methods to significant social or
behavioral issues, in such a way so as to be accessible to a nontechnical- oriented
readership (e.g., non-methodological researchers, teachers, students, govern-
ment personnel, practitioners, and other professionals). Applications from
a variety of disciplines such as psychology, public health, sociology, educa-
tion, and business are welcome. Books can be single- or multiple-authored
or edited volumes that (1) demonstrate the application of a variety of multi-
variate methods to a single, major area of research; (2) describe a multivariate
procedure or framework that could be applied to a number of research areas;
or (3) present a variety of perspectives on a topic of interest to applied multi-
variate researchers.
There are currently 20 books in the series:
• What If There Were No Significance Tests? co-edited by Lisa L. Harlow,
Stanley A. Mulaik, and James H. Steiger (1997)
• Structural Equation Modeling with LISREL, PRELIS, and SIMPLIS: Basic
Concepts, Applications, and Programming, written by Barbara M. Byrne

(1998)
• Multivariate Applications in Substance Use Research: New Methods for
New Questions, co-edited by Jennifer S. Rose, Laurie Chassin, Clark
C. Presson, and Steven J. Sherman (2000)
• Item Response Theory for Psychologists, co-authored by Susan E.
Embretson and Steven P. Reise (2000)
• Structural Equation Modeling with AMOS: Basic Concepts, Applications,
and Programming, written by Barbara M. Byrne (2001)
• Conducting Meta-Analysis Using SAS, written by Winfred Arthur, Jr.,
Winston Bennett, Jr., and Allen I. Huffcutt (2001)
• Modeling Intraindividual Variability with Repeated Measures Data:
Methods and Applications, co-edited by D. S. Moskowitz and Scott L.
Hershberger (2002)
• Multilevel Modeling: Methodological Advances, Issues, and Applications,
co-edited by Steven P. Reise and Naihua Duan (2003)
• The Essence of Multivariate Thinking: Basic Themes and Methods, written
by Lisa Harlow (2005)
• Contemporary Psychometrics: A Festschrift for Roderick P. McDonald,
co-edited by Albert Maydeu-Olivares and John J. McArdle (2005)
• Structural Equation Modeling with EQS: Basic Concepts, Applications, and
Programming, Second Edition, written by Barbara M. Byrne (2006)
• A Paul Meehl Reader: Essays on the Practice of Scientific Psychology,
co-edited by Niels G. Waller, Leslie J. Yonce, William M. Grove,
David Faust, and Mark F. Lenzenweger (2006)
• Introduction to Statistical Mediation Analysis, written by David P.
MacKinnon (2008)
• Applied Data Analytic Techniques for Turning Points Research, edited by
Patricia Cohen (2008)
• Cognitive Assessment: An Introduction to the Rule Space Method, written
by Kikumi K. Tatsuoka (2009)
• Structural Equation Modeling with AMOS: Basic Concepts, Applications,
and Programming, Second Edition written by Barbara M. Byrne (2010)
• Handbook of Ethics in Quantitative Methodology, co-edited by Abigail T.
Panter and Sonya K. Sterba (2011)
• Longitudinal Data Analysis: A Practical Guide for Researchers in Aging,
Health, and Social Sciences, co-edited by Jason T. Newsom, Richard N.
Jones, and Scott M. Hofer (2011)
• Structural Equation Modeling with MPlus: Basic Concepts, Applications,
and Programming written by Barbara M. Byrne (2012)

• Understanding The New Statistics: Effect Sizes, Confidence Intervals, and
Meta-Analysis, written by Geoff Cumming (2012)
Anyone wishing to submit a book proposal should send the following:
(1) author/title; (2) timeline including completion date; (3) brief overview of
the book’s focus, including table of contents and, ideally, a sample chapter
(or chapters); (4) a brief description of competing publications; and (5) tar-
geted audiences.
For more information, please contact the series editor, Lisa Harlow, at
Department of Psychology, University of Rhode Island, 10 Chafee Road,
Suite 8, Kingston, RI 02881-0808; phone [phone]; fax [phone]; or
e-mail [email protected]. Information may also be obtained from members of
the editorial/advisory board: Leona Aiken (Arizona State University), Daniel
Bauer (University of North Carolina), Jeremy Biesanz (University of British
Columbia), Gwyneth Boodoo (Educational Testing Services), Barbara M.
Byrne (University of Ottawa), Scott Maxwell (University of Notre Dame), Liora
Schmelkin (Hofstra University), and Stephen West (Arizona State University).
Front cover: Detail is from Treasure by Claire Layman, and is used with her generous permission.
Routledge Routledge
Taylor & Francis Group Taylor & Francis Group
711 Third Avenue 27 Church Road
New York, NY 10017 Hove, East Sussex BN3 2FA
© 2012 by Taylor & Francis Group, LLC
Routledge is an imprint of Taylor & Francis Group, an Informa business
Printed in the United States of America on acid-free paper
Version Date: 20110518
International Standard Book Number: 978-0-415-87967-5 (Hardback) 978-0-415-87968-2 (Paperback)
For permission to photocopy or use material electronically from this work, please access www.
copyright.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc.
(CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organiza-
tion that provides licenses and registration for a variety of users. For organizations that have been
granted a photocopy license by the CCC, a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and
are used only for identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the Psychology Press Web site at
http://www.psypress.com
For Lindy

Contents
Preface .................................................................................................................ix
1. Introduction to The New Statistics ........................................................1
2. From Null Hypothesis Significance Testing to Effect Sizes ...........21
3. Confidence Intervals ...............................................................................53
4. Confidence Intervals, Error Bars, and p Values .................................87
5. Replication ...............................................................................................119
6. Two Simple Designs..............................................................................153
7. Meta-Analysis 1: Introduction and Forest Plots ..............................181
8. Meta-Analysis 2: Models ......................................................................207
9. Meta-Analysis 3: Larger-Scale Analyses ..........................................231

10. The Noncentral t Distribution ............................................................263
11. Cohen’s d ..................................................................................................281
12. Power ........................................................................................................321
13. Precision for Planning ..........................................................................355
14. Correlations, Proportions, and Further Effect Size Measures .....381
15. More Complex Designs and The New Statistics in Practice..........411
Glossary ..........................................................................................................439
Commentary on Selected Exercises ..........................................................447
Appendices .....................................................................................................477
References ......................................................................................................495
Copyright Permissions ................................................................................503
Author Index ..................................................................................................505
Subject Index ..................................................................................................511
vii

Preface
The “new statistics” of this book’s title are not themselves new, but adopt-
ing them as the main way to analyze data would, for many students
and researchers, be new. It would also be an excellent development. Let
me explain.
“Children in the new program made significantly greater gains, p < .05.”
In many disciplines, that’s the standard way to express a conclusion. It
relies on null hypothesis significance testing (NHST), which uses p < .05 or
p < .01 to establish a result as statistically significant. However, NHST is a
very limited way to analyze and interpret data, because it focuses on the
narrow question, “Is there an effect?” Other disciplines, including physics
and chemistry, seldom use NHST, and usually report results as estimates
by saying, for example: “the melting point of the plastic was 85.5 ± 0.2°C.”
That’s usually much more informative than a statement that an effect is,
or is not, statistically significant. The main message of this book is that we
should shift emphasis as much as possible from NHST to estimation, based
on effect sizes and confidence intervals. We should to some extent join
physics, chemistry, and other disciplines that make enormous scientific
progress with little use of NHST.

For more than 50 years, scholars have explained the limitations of NHST
and the advantages of estimation and other preferred techniques, yet in
many disciplines we still feel obliged to use NHST. If we can progressively
free ourselves from that obligation, I suspect we’ll find that what I’m call-
ing “the new statistics” match the way we naturally think of results and
the interpretation of research. I don’t underestimate the challenge of over-
coming years of relying on p < .05, but once we are using the new statistics
I suspect we’ll find that they “feel right.” I’ll also discuss easy ways to
translate between the new and the old.
Effect sizes and confidence intervals provide more complete informa-
tion than does NHST. Meta-analysis allows accumulation of evidence
over a number of studies. Using these new statistics techniques, students
and researchers will be better informed and science will progress faster.
“Understanding” in the book’s title defines two goals: the “why” and the
“what.” The first is understanding why the new statistics are better and
why it’s worth the effort to change, and the second is understanding the
techniques themselves and how to use them in practice.
“It’s hard to change, do I really have to?” “Why now?” In reply I can say
that, first, change is already happening. Many journal editors are requir-
ing effect sizes, more articles that report confidence intervals are being
ix