Practical Meta-Analysis PDF
Preview Practical Meta-Analysis
Cover Page i PRACTICAL METAANALYSIS Applied Social Research Methods Series Volume 49 Page ii APPLIED SOCIAL RESEARCH METHODS SERIES Series Editors LEONARD BICKMAN, Peabody College, Vanderbilt University, Nashville DEBRA J. ROG, Vanderbilt University, Washington, DC 1. SURVEY RESEARCH METHODS 16. INTERPRETIVE INTERACTIONISM Second Edition by NORMAN K. DENZIN by FLOYD J. FOWLER, Jr. 17. ETHNOGRAPHY 2. INTEGRATING RESEARCH by DAVID M. FETTERMAN Second Edition by HARRIS M. COOPER 18. STANDARDIZED SURVEY INTERVIEWING by FLOYD J. FOWLER, Jr. 3. METHODS FOR POLICY RESEARCH and THOMAS W. MANGIONE by ANN MAJCHRZAK 19. PRODUCTIVITY MEASUREMENT 4. SECONDARY RESEARCH by ROBERT O. BRINKERHOFF Second Edition and DENNIS E. DRESSLER by DAVID W. STEWART and MICHAEL A. KAMINS 20. FOCUS GROUPS by DAVID W. STEWART 5. CASE STUDY RESEARCH and PREM N. SHAMDASANI Second Edition by ROBERT K. YIN 21. PRACTICAL SAMPLING by GARY T. HENRY 6. METAANALYTIC PROCEDURES FOR SOCIAL RESEARCH 22. DECISION RESEARCH Revised Edition by JOHN S. CARROLL by ROBERT ROSENTHAL and ERIC J. JOHNSON 7. TELEPHONE SURVEY METHODS 23. RESEARCH WITH HISPANIC POPULATIONS Second Edition by GERARDO MARIN by PAUL J. LAVRAKAS and BARBARA VANOSS MARIN 8. DIAGNOSING ORGANIZATIONS 24. INTERNAL EVALUATION Second Edition by ARNOLD J. LOVE by MICHAEL I. HARRISON 25. COMPUTER SIMULATION APPLICATIONS 9. GROUP TECHNIQUES FOR IDEA BUILDING by MARCIA LYNN WHICKER Second Edition and LEE SIGELMAN by CARL M. MOORE 26. SCALE DEVELOPMENT 10. NEED ANALYSIS by ROBERT F. DeVELLIS by JACK McKILLIP 27. STUDYING FAMILIES 11. LINKING AUDITING AND METAEVALUATION by ANNE P. COPELAND by THOMAS A. SCHWANDT and KATHLEEN M. WHITE and EDWARD S. HALPERN 28. EVENT HISTORY ANALYSIS 12. ETHICS AND VALUES IN APPLIED SOCIAL by KAZUO YAMAGUCHI RESEARCH by ALLAN J. KIMMEL 29. RESEARCH IN EDUCATIONAL SETTINGS by GEOFFREY MARUYAMA 13. ON TIME AND METHOD and STANLEY DENO by JANICE R. KELLY and JOSEPH E. McGRATH 30. RESEARCHING PERSONS WITH MENTAL ILLNESS by ROSALIND J. DWORKIN 14. RESEARCH IN HEALTH CARE SETTINGS by KATHLEEN E. GRADY 31. PLANNING ETHICALLY RESPONSIBLE RESEARCH and BARBARA STRUDLER WALLSTON by JOAN E. SIEBER 15. PARTICIPANT OBSERVATION by DANNY L. JORGENSEN Other volumes in this series are listed at the end of the book Page iii PRACTICAL METAANALYSIS Mark W. Lipsey David B. Wilson Applied Social Research Methods Series Volume 49 Page iv Copyright © 2001 by Sage Publications, Inc. All rights reserved. No part of this book may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from the publisher. For information address: SAGE Publications, Inc. 2455 Teller Road Thousand Oaks, California 91320 Email: [email protected] SAGE Publications Ltd. 6 Bonhill Street London EC2A 4PU United Kingdom SAGE Publications India Pvt. Ltd. M32 Market Greater Kailash I New Delhi 110 048 India Printed in the United States of America Library of Congress CataloginginPublication Data Lipsey, Mark W. Practical metaanalysis / by Mark W. Lipsey & David B. Wilson. p. cm. — (Applied social research methods series ; v. 49) Includes bibliographical references and index. ISBN 0761921672 (alk. paper). — ISBN 0761921680 (pbk. : alk. paper) 1. Metaanalysis. 2. Social sciences—Statistical methods. I. Wilson, David B., 1961– II. Title. III Series. HA29 .L83153 2000 300'.7'2—dc21 00035379 This book is printed on acidfree paper. 01 10 9 8 7 6 5 4 3 2 1 Acquiring Editor: C. Deborah Laughton Editorial Assistant: Eileen Carr Sage Production Editor: Sanford Robinson Typesetter: Technical Typesetting Inc. Page v Contents Acknowledgments ix 1. Introduction 1 Situations to Which MetaAnalysis is Applicable 1 The Key Concept of Effect Size 3 The Strengths of MetaAnalysis 5 The Weaknesses of MetaAnalysis 7 Recent History and Contemporary Usage of MetaAnalysis 10 Overview of Book 11 2. Problem Specification and Study Retrieval 12 Identifying the Form of the Research Findings to be MetaAnalyzed 12 Study Eligibility Criteria 16 Methodological Quality Revisited 20 Identifying, Locating, and Retrieving Research Reports 23 Finding References 24 Retrieving Research Reports 31 3. Selecting, Computing, and Coding the Effect Size Statistic 34 Effect Size Statistics and Their Variances 34 A Note on Notation 37 Types of Research Findings and Applicable Effect Size Statistics 37 OneVariable Relationships (Central Tendency Description) 38 TwoVariable Relationships 41 Association Between Variables 59 Multivariate Relationships 67 Summary of Effect Size Statistics 71 4. Developing a Coding Scheme and Coding Study Reports 73 Developing a Coding Protocol 73 Units of Analysis and Hierarchical Levels of Coding 75 Effect Size Coding 81 Study Descriptors 83 Page vi Coding Information About the Coding Process Itself 86 Training of Coders 88 5. Data Management 91 Creating MetaAnalytic Data Files 91 Coding Directly into the Computer 94 Using the Computer to Maintain the Bibliography 95 Structure of MetaAnalytic Data Files 96 Creating a Single Flat File to Use for Analysis 97 Merging Multiple Files to Use for Analysis 97 6. Analysis Issues and Strategies 105 The Stages of Analysis 107 Effect Size Adjustments 107 Analyzing the Effect Size Mean and Distribution 112 Analysis of Heterogeneous Distributions of Effect Size 116 Analysis of Statistically Dependent Effect Sizes 125 Additional Analysis Issues 126 7. Computational Techniques for MetaAnalysis Data 129 The Mean, Confidence Interval, and Homogeneity Test 129 Analysis of Heterogeneous Distributions of Effect Size 133 Weighted Regression Analysis 138 Graphing Techniques 142 8. Interpreting and Using MetaAnalysis Results 146 Interpreting Effect Size Values 146 Rules of Thumb for Effect Size Magnitude 147 Translation of Effect Sizes to Other Metrics 148 Clinical and Practical Significance 154 Caveats in Interpreting MetaAnalysis Results 156 Methodological Adequacy of the Research Base 157 Confounding of Substantive and Methodological Features 158 The Importance of Variance 161 Research Gaps and Generalizability 163 Sampling Bias 165 Implications of MetaAnalysis for Practice and Policy 167 Appendix A. ComputerBased Bibliographic Services and Examples of Relevant Databases 169 Page vii Appendix B. Procedures for Computing Effect Size Values from Eligible Study Reports 172 Standardized Mean Difference Effect Size 172 Direct Calculation of ESsm 172 Algebraically Equivalent Formulas for ESsm 173 Exact Probability Levels for a tvalue or Fratio 174 Calculation of Means and Standard Deviations from a Frequency Distribution 175 Approximations Based on Continuous Data—The PointBiserial Coefficient 177 Estimating and spooled 178 Dichotomized Data 187 The Correlation Coefficient Effect Size 189 Definitional Formula for ESr 190 Joint Frequency Distributions for Discrete or Grouped Continuous Data 191 A Dichotomous and a Continuous Measure 192 Two Dichotomous Measures 194 Approximations and Probability Values 195 OddsRatio Effect Size 195 Calculation Based on Cell Frequencies 196 Calculation Based on Row Proportions 196 Calculation Based on Cell Proportions 196 Imputation of 2 × 2 Contingency Table From Correlation and Marginal Proportions 197 Imputation of 2 × 2 Contingency Table Based on ChiSquare and Marginal Proportions 197 Imputation of OddsRatio from Continuous Data 198 Appendix C. MS Excel Effect Size Computation Program 207 Appendix D. SPSS Macros for MetaAnalysis 208 Appendix E. Coding Manual and Coding Forms for the Example MetaAnalysis of Challenge Programs for 221 Juvenile Delinquents StudyLevel Coding Manual 221 Sample Descriptors 221 Research Design Descriptors 222 Nature of the Treatment Descriptors 223 Page viii Effect Size Level Coding Manual 225 Dependent Measure Descriptors 225 Effect Size Data 226 StudyLevel Coding Form 228 Sample Descriptors 228 Research Design Descriptors 229 Nature of the Treatment Descriptors 230 Effect Size Level Coding Form 231 Dependent Measure Descriptors 231 Effect Size Data 231 Bibliography of Recommended Readings 233 References 234 Index 241 About the Authors 247 Page ix Acknowledgments We thank Steve Leff and his colleagues at HSRI for stimulating the initial metaanalysis "toolkit" that has evolved into this volume and for helping improve the final product. Our appreciation also goes to Sandra Jo Wilson for generously sharing the juvenile delinquency challenge metaanalysis used as a running example throughout this book, to Will Shadish for his valuable suggestions and encouragement as the manuscript progressed, to Jim Derzon for his spirited partnership in the many meta analysis projects that sharpened our understanding of the craft, and to Laurie Samuels for her assiduous work on the index. Finally, to the many students who labored through draft versions in our workshops and seminars, we offer a special thanks for your patience and tactful editorial assistance.
The list of books you might like
