Examining the advantages and disadvantages of pilot studies PDF
Preview Examining the advantages and disadvantages of pilot studies
University of New Mexico UNM Digital Repository Psychology ETDs Electronic Theses and Dissertations 2-1-2012 Examining the advantages and disadvantages of pilot studies : Monte-Carlo simulations Masato Nakazawa Follow this and additional works at:https://digitalrepository.unm.edu/psy_etds Recommended Citation Nakazawa, Masato. "Examining the advantages and disadvantages of pilot studies : Monte-Carlo simulations." (2012). https://digitalrepository.unm.edu/psy_etds/104 This Dissertation is brought to you for free and open access by the Electronic Theses and Dissertations at UNM Digital Repository. It has been accepted for inclusion in Psychology ETDs by an authorized administrator of UNM Digital Repository. For more information, please contact [email protected]. Examining the Advantages and Disadvantages of Pilot Studies: Monte- Carlo Simulations BY Masato Nakazawa B.S., History, the University of California, Los Angeles, 2000 M.A., Psychology, the University of New Mexico, 2006 DISSERTATION Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy Psychology The University of New Mexico Albuquerque, New Mexico December, 2011 ©2011, Masato Nakazawa iii ACKNOWLEDGMENTS I heartily acknowledge Dr. Harold Delaney, my advisor and dissertation chair, for continuing to encourage me through the years of classroom teachings and for his infinite patience in reading my manuscript and giving me feedback. His guidance and professionalism will remain with me as I continue my career. I also thank my committee members, Dr. Angela Bryan, Dr. Timothy Goldsmith, and Dr. Jay Parkes, for their valuable suggestions and encouragement that played an important role in shaping this study and my professional development. And finally to my wife, Yea-Wen Chen. You bestowed the greatest gifts of love and encouragement upon me. This dissertation is our accomplishment. iv Examining the Advantages and Disadvantages of Pilot Studies: Monte- Carlo Simulations BY Masato Nakazawa ABSTRACT OF DISSERTATION Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy Psychology The University of New Mexico Albuquerque, New Mexico December, 2011 Examining the Advantages and Disadvantages of Pilot Studies: Monte-Carlo Simulations By Masato Nakazawa B.A. in History, University of California, Los Angeles, 2000 M.S. in Psychology, University of New Mexico, 2006 Ph.D. in Psychology, University of New Mexico, 2011 Abstract Estimating population effect size accurately and precisely plays a vital role in achieving a desired level of statistical power as well as drawing correct conclusions from empirical results. While a number of common practices of effect-size estimation have been documented (e.g., relying on one’s experience and intuition, and conducting pilot studies), their relative advantages and disadvantages have been insufficiently investigated. To establish a practical guideline for researchers in this respect, this project compared the accuracy and precision of effect-size estimation, resulting power, and economic implications across pilot and non-pilot conditions. Furthermore, to model the potential advantages of conducting pilot studies in finding and correcting flaws before main studies are run, varying amounts of random error variance and varying degrees of vi success at its removal – often neglected aspects in simulation studies – were introduced in Experiment 2. The main findings include the following. First, pilot studies with up to 30 subjects were utterly ineffective in achieving the desired power of 0.80 at a small population effect size even under the best-case scenario. At this effect size, intuitive estimation without pilot studies appears to be the preferred method of achieving the desired power. Second, the pilot studies performed better at medium and large population effect sizes, achieving comparable or even greater power to that in the non-pilot condition. The relative advantages of pilot studies were particularly evident when moderate to large error variances were present, and a portion of it had been removed through conducting pilot studies. These broad findings are discussed in the context of flexible design: study design can be modified flexibly in accordance with the researcher’s particular goals. vii TABLE OF CONTENTS LIST OF FIGURES………………………………………………………………...… …xii LIST OF TABLES……………………………………………………………………... xiii CHAPTER 1 INTRODUCTION…………………………………………………….…… 1 Importance of Power…………………………………………………………………. 1 Practices of Effect-Size Estimation………………………………………………...… 5 CHAPTER 2 OBJECTIVES OF THE CURRENT STUDY……………………………. 11 Objective 1…………………………………………………………………………... 11 Objective 2…………………………………………………………………………... 11 Objective 3…………………………………………………………………………... 12 CHAPTER 3 GENERAL METHOD…………………………………………………….15 Procedure……………………………………………………………………………. 15 Independent Variables………………………………………………………………. 17 Dependent Variables………………………………………………………………... 18 Estimated required sample size………………………………………………….18 Power deviation………………………………………………………………….18 Measures of accuracy and precision of effect-size estimation…………………..19 Measures of economic performance…………………………………………….20 Cost per percentage point………………………………………………… 20 Expected wasted resources…………………………………………………21 CHAPTER 4 EXPERIMENT 1………………………………………………………… 23 Method……………………………………………………………………………… 23 Effect-size Estimation methods………………………………………………….23 Hedges formula………………………………………………………………25 viii Wherry formula……………………………………………………………. 25 Maxwell-Delaney (MD) formula…………………………………………... 25 Upper one-sided confidence limit (UCL)………………………………….. 26 Results……………………………………………………………………………….. 27 Observed effect sizes …………………………………………………27 Measures of accuracy of effect-size estimation…………………………... ..27 Overall impression……………………………………………………… .28 Cohen’s d………………………………………………………………... 28 Hedges formula………………………………………………………….. 31 Wherry formula…………………………………………………………. 31 Maxwell-Delaney (MD) formula………………………………………... 31 Upper one-sided confidence limit (UCL)……………………………….. 32 Measures of precision of effect-size estimation………………………..….. .32 Overall impression………………………………………………………. 32 Cohen’s d………………………………………………………………... 34 Hedges formula and UCL………………………………………………. 34 Wherry and MD formulae……………………………………………….. 35 Ninety-five percent confidence interval around observed effect size…………… 35 Estimated required sample size………………………………………………. ….38 Overall impression…………………………………………………………. 39 Cohen’s d…………………………………………………………………… 41 Hedges formula……………………………………………………………...41 Wherry and MD formulae…………………………………………………...42 UCL…………………………………………..……………………………...42 Probability of the main study being aborted…………………………………..…43 ix
Description:The list of books you might like
