Effect Size for ANOVA Designs PDF
Preview Effect Size for ANOVA Designs
Effect Size for Anova Designs SAGE UNIVERSITY PAPERS Series: Quantitative Applications in the Social Sciences Series Editor: Michael S. LewisBeck, University of Iowa Editorial Consultants Richard A. Berk, Sociology, University of California, Los Angeles William D. Berry, Political Science, Florida State University Kenneth A. Bollen, Sociology, University of North Carolina, Chapel Hill Linda B. Bourque, Public Health, University of California, Los Angeles Jacques A. Hagenaars, Social Sciences, Tilburg University Sally Jackson, Communications, University of Arizona Richard M. Jaeger, Education, University of North Carolina, Greensboro Gary King, Department of Government, Harvard University Roger E. Kirk, Psychology, Baylor University Helena Chmura Kraemer, Psychiatry and Behavioral Sciences, Stanford University Peter Marsden, Sociology, Harvard University Helmut Norpoth, Political Science, SUNY, Stony Brook Frank L. Schmidt, Management and Organization, University of Iowa Herbert Weisberg, Political Science, The Ohio State University Publisher Sara Miller McCune, Sage Publications, Inc. INSTRUCTIONS TO POTENTIAL CONTRIBUTORS For guidelines on submission of a monograph proposal to this series, please write Michael S. LewisBeck, Editor Sage QASS Series Department of Political Science University of Iowa Iowa City, IA 52242 Page i Series/Number 07129 Effect Size for Anova Designs Jose M. Cortina George Mason University Hossein Nouri College of New Jerseys SAGE PUBLICATIONS International Educational and Professional Publisher Thousand Oaks London New Delhi Page ii Copyright © 2000 by Sage Publications, Inc. All rights reserved. No part of this book may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from the publisher. For information: SAGE Publications, Inc. 2455 Teller Road Thousand Oaks, California 91320 Email: [email protected] SAGE Publications Ltd. 6 Bonhill Street London EC2A 4PU United Kingdom SAGE Publications India Pvt. Ltd. M32 Market Greater Kailash I New Delhi 110 048 India Printed in the United States of America Library of Congress CataloginginPublication Data Cortina, Jose M. Effect size for ANOVA designs / by Jose M. Cortina and Hossein Nouri. p. cm. — (Sage university papers series. Quantitative applications in the social sciences; 07129) Includes bibliographical references. ISBN 0761915508 (acidfree paper) 1. Experimental design. 2. Analysis of variance. I. Nouri, Hossein. II. Title. III. Series: Sage university papers series. Quantitative applications in the social sciences ; no. 07129. QA279.C6645 1999 519.5'38—dc21 996532 00 01 02 03 04 05 06 7 6 5 4 3 2 1 When citing a university paper, please use the proper form. Remember to cite the Sage University Paper series title and include the paper number. One of the following formats can be adapted (depending on the style manual used): (1) CORTINA, JOSE M. (2000) Effect size for ANOVA designs. Sage University Papers Series on Quantitative Applications in the Social Sciences, 07129. Thousand Oaks, CA: Sage. OR (2) Cortina, J. M. (2000). Effect size for ANOVA designs. (Sage University Papers Series on Quantitative Applications in the Social Sciences, series no. 07129). Thousand Oaks, CA: Sage. Page iii Contents Series Editor's Introduction v Introduction 1 The Data Set 3 1. The Two Independent Group Design: A Review 4 Estimation of Sampling Error Variance 8 Summary 10 2. OneWay Independent Groups Design 10 Summary 14 3. Factorial Independent Group Design 14 2 × 2 Analysis of Variance (ANOVA) 14 Summary 39 4. ANCOVA Designs 40 Effect Size Holding the Covariate Constant 42 Other ANCOVA Designs 45 HigherOrder ANCOVAs 46 Summary 48 5. Repeated Measures Designs 49 HigherOrder Repeated Measures Designs 52 Computing Sampling Error Variance of Effect Sizes 55 Summary 56 6. Monograph Summary and Discussion of Ancillary Issues 56 Summary 56 Ancillary Issues 57 Notes 63 Appendix A: Tables 64 Appendix B: Derivation for Equation (4.7) 71 References 72 About the Authors 73 Acknowledgments 74 Page v Series Editor's Introduction How big is the effect of an independent variable? That may well be the principal question a researcher asks, at least once it is established that the effect is significantly different from zero. Even with the simplest of designs, however, it may not be easy to assess the magnitude of an effect. Suppose an education psychologist is assessing reading speed in a two independent group study, where there are 50 randomly assigned college sophomores in each. Group A reads a guide to improving reading speed, whereas Group B receives a lecture on speed reading. After exposure to these treatments, subjects take a multitude item test measuring comprehended words (CW). For Group A the mean CW = 54.6, and for Group B the mean CW = 65.1. The difference of 11.5 is found to be statistically significant at .05, suggesting that the lecture improved reading speed. But how big is that effect? Is 11.5 a "big number?" We cannot say unless we know more. In particular, we need to know the spread of scores of the reading speed variable. On the one hand, if in the population CW is normally distributed and goes from 2.1 to 306.7, then the effect—11.5— covers only a little part of the range and must appear small. On the other hand, if that range is from 2.1 to 30.6, then the effect appears large. One approach to this variability problem is standardization of the effect measure. The authors emphasize the use of "d," which essentially divides the observed effect by the standard deviation of the dependent variable. In our example, we would divide the mean difference (11.5) by the pooled standard deviation of CW (28.3), yielding d = .41. This suggests that the effects of the instructional change is moderate, a conclusion reinforced by the conversion of d to a Pearson's r of .20. (The general conversion formula is provided therein). Calculating d can become complicated with more complex designs. Fortunately, in this single source Dr. Cortina and Dr. Nouri carefully layout the computational methods for d with a variety of designs, Page vi including factorial ANOVA, ANCOVA, and repeated measures ANOVA. Also, they develop rules for the "secondhand" calculations needed to arrive at d values from other studies, necessary in order to carry out metaanalyses. Throughout, the computations are illustrated on the same data, a comprehensive set of simulated observations on air traffic controllers. The dependent variable is always a task performance, related to different independent variables, and showing effect size calculation under the different ANOVA designs. While the text throughly covers computational steps, it goes beyond them to raise important theoretical issues. An innovative discussion of the place of "offfactors," in arriving at effect size, is given. Also, in the closing chapter, important questions of research practice are raised. How different are the different measures of d? What about changing variance in the independent variables? In a metaanalysis, should one rely on d or r? Is unequal subgroup size a problem? Should research reporting guidelines be altered? These are all issues that the "thinking practioner" must weigh when actually writing up and interpreting his or her own results, from an assessment of effect size based on the computational methods in this excellent handbook. —MICHAEL S. LEWISBECK SERIES EDITOR Page 1 Introduction There was a time, not long ago, when we social scientists relied almost entirely on tests of statistical significance to provide us with the information that we needed to draw our conclusions. The significance test is not without its advantages (Cortina & Dunlap, 1997), but it also suffers from certain limitations. The limitation of primary interest here is that tests of statistical significance are affected greatly by sample size. Thus, for example, a very weak effect can be "statistically significant" while a fairly strong effect can fail to attain statistically significance (see Meehl, 1990 for examples of the former, and Hunter & Schmidt, 1990 for examples of the latter). Due to these interpretational limitations, we often want information relating not only to the question of whether or not an effect exists, but also to the magnitude of the effect. This is typically accomplished by standardizing some index of raw magnitude (e.g., a difference between group means) so that variability is held constant across variables or studies. The result is an index such as d, r2, or 2. For many research designs, the computation of standardized magnitude indices is quite simple. For example, if we employ a twogroup design, we can compute d by finding the difference between the group means and dividing by the pooled or control group standard deviation. Moreover, this value gives us an index of magnitude of effect for both the independent groups and repeated measures designs. If instead we employ a simple correlational design with continuous variables, then the Pearson correlation coefficient provides an analogous index. Of course, the pointbiserial correlation coefficient, d, F , and t are easily converted into one another 1, N1 + N2 – 2 N1 + N2 – 2 Page 2 with readily available formulas (see our Table 2.1 or Hunter & Schmidt, 1990, p. 273). 1 For more complicated designs, computation of standardized magnitude indices is not necessarily so simple. If, for example, we were to take the twogroup design and add a covariate such that we wanted an index of magnitude of effect with the covariate held constant, computation is, as we see later in this monograph, no longer straightforward. These more complicated designs also make secondhand computation of magnitude indices more difficult. By secondhand, we mean computations, typical of meta analysis, that must be performed on results as reported in written work. Suppose that we are conducting a metaanalysis of the relationship between mental practice and task performance. Some of the studies from which we would like to extract information might have employed twogroup, repeated measures designs such that task performance was measured for each participant after mental practice and after no practice. If these studies reported group means and standard deviations, then d values are easily calculated using the formula described above. Oftentimes, however, effect size values must be computed from test statistics such as t values. In order to convert a repeated measures t value into a d value, we must rely on the authors of these studies to provide us with descriptive information such as the measure to measure correlations. We must also know how to use this information to generate the values that we need. As researchers often find themselves facing such complex designs, there exists a need for clarification regarding the methods that might be used to compute effect size values from such designs. To this end, the purpose of this monograph is to describe methods for computing indices of standardized magnitude of effect from complex ANOVA designs. By "complex" we simply mean designs that involve repeated measurements, multiple independent variables, or covariates. Thus, we focus on repeated measures ANOVA, factorial ANOVA, and ANCOVA designs. MANOVA, MANCOVA, and correlational designs are beyond the scope of this manuscript. As is true of most or all of the QASS monographs, much (though not all) of the material in the present monograph has been presented before, if not by one of the present authors, then by others. It is our hope that this monograph can serve as a single resource that accommodates a variety of needs viz. effect size computation for ANOVA
Description:The list of books you might like
