Table Of Content

WWeesstteerrnn MMiicchhiiggaann UUnniivveerrssiittyy SScchhoollaarrWWoorrkkss aatt WWMMUU Dissertations Graduate College 1-2011 AAnn AAnnaalloogg EExxppeerriimmeenntt CCoommppaarriinngg GGooaall--FFrreeee EEvvaalluuaattiioonn aanndd GGooaall AAcchhiieevveemmeenntt EEvvaalluuaattiioonn UUttiilliittyy Brandon W. Youker Western Michigan University Follow this and additional works at: https://scholarworks.wmich.edu/dissertations Part of the Policy Design, Analysis, and Evaluation Commons, Social Statistics Commons, and the Statistics and Probability Commons RReeccoommmmeennddeedd CCiittaattiioonn Youker, Brandon W., "An Analog Experiment Comparing Goal-Free Evaluation and Goal Achievement Evaluation Utility" (2011). Dissertations. 486. https://scholarworks.wmich.edu/dissertations/486 This Dissertation-Open Access is brought to you for free and open access by the Graduate College at ScholarWorks at WMU. It has been accepted for inclusion in Dissertations by an authorized administrator of ScholarWorks at WMU. For more information, please contact [email protected]. AN ANALOG EXPERIMENT COMPARING GOAL-FREE EVALUATION AND GOAL ACHIEVEMENT EVALUATION UTILITY by Brandon W. Youker A Dissertation Submitted to the Faculty of The Graduate College in partial fulfillment of the requirements for the Degree of Doctor of Philosophy Interdisciplinary Ph.D. in Evaluation Advisor: Chris L. S. Coryn, Ph.D. Western Michigan University Kalamazoo, Michigan December 2011 AN ANALOG EXPERIMENT COMPARING GOAL-FREE EVALUATION AND GOAL ACHIEVEMENT EVALUATION UTILITY Brandon W. Youker, Ph.D. Western Michigan University, 2011 Goal-free evaluation (GFE) is the process of determining the merit of an evaluand independent of the stated or implied goals and objectives, whereas goal achievement evaluation (GAE), as the most rudimentary form of goal-based evaluation, determines merit according to the evaluand’s level of accomplishment with regard to its goals. This study examines the utility of GAE and GFE from the perspective of the evaluation’s intended users. In the study, two evaluation teams, goal achievement and goal-free, independently and simultaneously evaluate the same human service program. Each team produced a final evaluation report, which was read by the evaluation’s users, who then responded to questionnaires regarding the reports’ usefulness and later interviews. The questionnaire results were that 66% of evaluation users scored GAE more favorably versus 33% who scored GFE higher. The results of the interviews were that 40% of evaluation users found GAE more useful, with 20% claiming GFE more useful; the remaining users were undecided or felt the approaches equal. The conclusion is that differences between the two evaluation reports exist; however, it is not apparent as to whether these differences are caused by implementing GAE or GFE. Furthermore, the effects or differences that did present between the evaluations were small and not practically significant enough to definitively claim one approach clearly more useful to these evaluation users. UMI Number: 3496368 All rights reserved INFORMATION TO ALL USERS The quality of this reproduction is dependent on the quality of the copy submitted. In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted. Also, if material had to be removed, a note will indicate the deletion. UMI 3496368 Copyright 2012 by ProQuest LLC. All rights reserved. This edition of the work is protected against unauthorized copying under Title 17, United States Code. ProQuest LLC. 789 East Eisenhower Parkway P.O. Box 1346 Ann Arbor, MI 48106 - 1346 Copyright by Brandon W. Youker 2011 ACKNOWLEDGMENTS The development of this dissertation has benefited from both the advice and criticisms of Professors Chris L. S. Coryn, David Hartmann, Liliana Rodríguez-Campos, Michael Scriven, James Sanders, and Michael Q. Patton. Further assistance was provided by P. Cristian Guigu, Amy Gullickson, Wes Martz, Daniela Schröeter, Lori Wingate, Sally Veeder, and Mary Ramlow. I would also like to recognize the challenges and opportunities provided by the Evaluation Center at Western Michigan University. Appreciation and recognition is due to the partnering organizations who allowed their program to be evaluated and their staff to participate, and who participated themselves. Lastly, a heartfelt thanks goes to my family, who has supported me academically, professionally, and personally. In particular, thank you to my parents, Bruce W. Youker and Christine A. Youker, and my father-in-law, David K. Caldwell. Finally, the love and encouragement of my wife, Elizabeth, and daughter, Vera, continue to motivate me both in my scholarly pursuits and, more importantly, in life. Brandon W. Youker ii TABLE OF CONTENTS ACKNOWLEDGMENTS.............................................................................................. ii LIST OF TABLES.......................................................................................................... viii LIST OF FIGURES........................................................................................................ xi CHAPTER I. INTRODUCTION TO THE PROBLEM............................................................ 1 Background of the Problem........................................................................... 1 Statement of the Problem Situation.............................................................. 5 Purpose of the Study..................................................................................... 5 Objectives to Be Investigated....................................................................... 7 Conceptual and Substantive Assumptions.................................................... 7 Assumption#1: Evaluation Utility Is Worthy of Study........................... 7 Assumption#2: GAE Is a Commonly Used Evaluation Approach......... 9 Analog Studies.............................................................................................. 10 Fidelity.......................................................................................................... 12 Statement of Hypotheses............................................................................... 13 Importance of the Study................................................................................ 14 Chapter Summary......................................................................................... 15 Outline of the Dissertation............................................................................ 15 II. LITERATURE REVIEW................................................................................... 16 Previous Empirical Studies on GFE............................................................. 16 iii Table of Contents—Continued CHAPTER Evaluation Utility.......................................................................................... 17 Current Study................................................................................................ 23 The History of GFE...................................................................................... 26 Prehistory................................................................................................ 27 Ancient History....................................................................................... 28 The European Renaissance..................................................................... 32 Tylerian Evaluation (Goal-Based Evaluation)........................................ 32 The Consumers Union............................................................................ 34 Contemporary Professional Evaluation.................................................. 36 The Logic of GFE......................................................................................... 39 Definition of GFE................................................................................... 39 Nature of GFE’s Relationships..................................................................... 53 Goal-Based Evaluation Principles.......................................................... 53 The Philosophy of GFE.......................................................................... 57 Fields of Evaluation................................................................................ 59 Rules of Inference Governing GFE........................................................ 63 Criticisms of GFE and Responses .......................................................... 98 Chapter Summary......................................................................................... 110 III. METHODOLOGY............................................................................................. 111 Description of the Approach......................................................................... 111 Research Design............................................................................................ 112 iv Table of Contents—Continued CHAPTER Subject Selection and Characteristics........................................................... 113 Evaluand Selection and Characteristics.................................................. 113 Evaluation User Selection and Characteristics....................................... 116 Evaluator Selection and Characteristics.................................................. 117 Study Setting................................................................................................. 121 Instrumentation and Materials...................................................................... 122 Evaluand-Created Materials.................................................................... 122 Investigator-Created Materials................................................................ 122 Evaluator-Created Materials................................................................... 125 Instruments.................................................................................................... 126 Utility Measures...................................................................................... 127 Procedures ..................................................................................................... 136 Phase One – Pre-Evaluation Phase......................................................... 138 Phase Two – Evaluation Phase............................................................... 139 Phase Three – Utility Study Phase.......................................................... 139 Phase Four – Utility Analysis and Reporting Phase............................... 139 Data Collection and Recording..................................................................... 140 Data Processing and Analysis....................................................................... 140 Methodological Limitations.......................................................................... 141 Chapter Summary......................................................................................... 143 v

Description:
this approach the evaluator's role was thought to be simply to test fact-based claims that originated in statements about program or policy goals; the
ebook img

An Analog Experiment Comparing Goal-Free Evaluation and Goal Achievement Evaluation Utility PDF

290 Pages·2017·1.78 MB·English
by  Brandon W. Youker
Save to my drive
Quick download
Download
Upgrade Premium
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.

Preview An Analog Experiment Comparing Goal-Free Evaluation and Goal Achievement Evaluation Utility

WWeesstteerrnn MMiicchhiiggaann UUnniivveerrssiittyy SScchhoollaarrWWoorrkkss aatt WWMMUU Dissertations Graduate College 1-2011 AAnn AAnnaalloogg EExxppeerriimmeenntt CCoommppaarriinngg GGooaall--FFrreeee EEvvaalluuaattiioonn aanndd GGooaall AAcchhiieevveemmeenntt EEvvaalluuaattiioonn UUttiilliittyy Brandon W. Youker Western Michigan University Follow this and additional works at: https://scholarworks.wmich.edu/dissertations Part of the Policy Design, Analysis, and Evaluation Commons, Social Statistics Commons, and the Statistics and Probability Commons RReeccoommmmeennddeedd CCiittaattiioonn Youker, Brandon W., "An Analog Experiment Comparing Goal-Free Evaluation and Goal Achievement Evaluation Utility" (2011). Dissertations. 486. https://scholarworks.wmich.edu/dissertations/486 This Dissertation-Open Access is brought to you for free and open access by the Graduate College at ScholarWorks at WMU. It has been accepted for inclusion in Dissertations by an authorized administrator of ScholarWorks at WMU. For more information, please contact [email protected]. AN ANALOG EXPERIMENT COMPARING GOAL-FREE EVALUATION AND GOAL ACHIEVEMENT EVALUATION UTILITY by Brandon W. Youker A Dissertation Submitted to the Faculty of The Graduate College in partial fulfillment of the requirements for the Degree of Doctor of Philosophy Interdisciplinary Ph.D. in Evaluation Advisor: Chris L. S. Coryn, Ph.D. Western Michigan University Kalamazoo, Michigan December 2011 AN ANALOG EXPERIMENT COMPARING GOAL-FREE EVALUATION AND GOAL ACHIEVEMENT EVALUATION UTILITY Brandon W. Youker, Ph.D. Western Michigan University, 2011 Goal-free evaluation (GFE) is the process of determining the merit of an evaluand independent of the stated or implied goals and objectives, whereas goal achievement evaluation (GAE), as the most rudimentary form of goal-based evaluation, determines merit according to the evaluand’s level of accomplishment with regard to its goals. This study examines the utility of GAE and GFE from the perspective of the evaluation’s intended users. In the study, two evaluation teams, goal achievement and goal-free, independently and simultaneously evaluate the same human service program. Each team produced a final evaluation report, which was read by the evaluation’s users, who then responded to questionnaires regarding the reports’ usefulness and later interviews. The questionnaire results were that 66% of evaluation users scored GAE more favorably versus 33% who scored GFE higher. The results of the interviews were that 40% of evaluation users found GAE more useful, with 20% claiming GFE more useful; the remaining users were undecided or felt the approaches equal. The conclusion is that differences between the two evaluation reports exist; however, it is not apparent as to whether these differences are caused by implementing GAE or GFE. Furthermore, the effects or differences that did present between the evaluations were small and not practically significant enough to definitively claim one approach clearly more useful to these evaluation users. UMI Number: 3496368 All rights reserved INFORMATION TO ALL USERS The quality of this reproduction is dependent on the quality of the copy submitted. In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted. Also, if material had to be removed, a note will indicate the deletion. UMI 3496368 Copyright 2012 by ProQuest LLC. All rights reserved. This edition of the work is protected against unauthorized copying under Title 17, United States Code. ProQuest LLC. 789 East Eisenhower Parkway P.O. Box 1346 Ann Arbor, MI 48106 - 1346 Copyright by Brandon W. Youker 2011 ACKNOWLEDGMENTS The development of this dissertation has benefited from both the advice and criticisms of Professors Chris L. S. Coryn, David Hartmann, Liliana Rodríguez-Campos, Michael Scriven, James Sanders, and Michael Q. Patton. Further assistance was provided by P. Cristian Guigu, Amy Gullickson, Wes Martz, Daniela Schröeter, Lori Wingate, Sally Veeder, and Mary Ramlow. I would also like to recognize the challenges and opportunities provided by the Evaluation Center at Western Michigan University. Appreciation and recognition is due to the partnering organizations who allowed their program to be evaluated and their staff to participate, and who participated themselves. Lastly, a heartfelt thanks goes to my family, who has supported me academically, professionally, and personally. In particular, thank you to my parents, Bruce W. Youker and Christine A. Youker, and my father-in-law, David K. Caldwell. Finally, the love and encouragement of my wife, Elizabeth, and daughter, Vera, continue to motivate me both in my scholarly pursuits and, more importantly, in life. Brandon W. Youker ii TABLE OF CONTENTS ACKNOWLEDGMENTS.............................................................................................. ii LIST OF TABLES.......................................................................................................... viii LIST OF FIGURES........................................................................................................ xi CHAPTER I. INTRODUCTION TO THE PROBLEM............................................................ 1 Background of the Problem........................................................................... 1 Statement of the Problem Situation.............................................................. 5 Purpose of the Study..................................................................................... 5 Objectives to Be Investigated....................................................................... 7 Conceptual and Substantive Assumptions.................................................... 7 Assumption#1: Evaluation Utility Is Worthy of Study........................... 7 Assumption#2: GAE Is a Commonly Used Evaluation Approach......... 9 Analog Studies.............................................................................................. 10 Fidelity.......................................................................................................... 12 Statement of Hypotheses............................................................................... 13 Importance of the Study................................................................................ 14 Chapter Summary......................................................................................... 15 Outline of the Dissertation............................................................................ 15 II. LITERATURE REVIEW................................................................................... 16 Previous Empirical Studies on GFE............................................................. 16 iii Table of Contents—Continued CHAPTER Evaluation Utility.......................................................................................... 17 Current Study................................................................................................ 23 The History of GFE...................................................................................... 26 Prehistory................................................................................................ 27 Ancient History....................................................................................... 28 The European Renaissance..................................................................... 32 Tylerian Evaluation (Goal-Based Evaluation)........................................ 32 The Consumers Union............................................................................ 34 Contemporary Professional Evaluation.................................................. 36 The Logic of GFE......................................................................................... 39 Definition of GFE................................................................................... 39 Nature of GFE’s Relationships..................................................................... 53 Goal-Based Evaluation Principles.......................................................... 53 The Philosophy of GFE.......................................................................... 57 Fields of Evaluation................................................................................ 59 Rules of Inference Governing GFE........................................................ 63 Criticisms of GFE and Responses .......................................................... 98 Chapter Summary......................................................................................... 110 III. METHODOLOGY............................................................................................. 111 Description of the Approach......................................................................... 111 Research Design............................................................................................ 112 iv Table of Contents—Continued CHAPTER Subject Selection and Characteristics........................................................... 113 Evaluand Selection and Characteristics.................................................. 113 Evaluation User Selection and Characteristics....................................... 116 Evaluator Selection and Characteristics.................................................. 117 Study Setting................................................................................................. 121 Instrumentation and Materials...................................................................... 122 Evaluand-Created Materials.................................................................... 122 Investigator-Created Materials................................................................ 122 Evaluator-Created Materials................................................................... 125 Instruments.................................................................................................... 126 Utility Measures...................................................................................... 127 Procedures ..................................................................................................... 136 Phase One – Pre-Evaluation Phase......................................................... 138 Phase Two – Evaluation Phase............................................................... 139 Phase Three – Utility Study Phase.......................................................... 139 Phase Four – Utility Analysis and Reporting Phase............................... 139 Data Collection and Recording..................................................................... 140 Data Processing and Analysis....................................................................... 140 Methodological Limitations.......................................................................... 141 Chapter Summary......................................................................................... 143 v

Description:
this approach the evaluator's role was thought to be simply to test fact-based claims that originated in statements about program or policy goals; the
See more

The list of books you might like

Upgrade Premium
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.
DMCA & Copyright: Dear all, most of the website is community built, users are uploading hundred of books everyday, which makes really hard for us to identify copyrighted material, please contact us if you want any material removed.
Copyright @ PDFdrive.to, 2022 | We love our users