Alternative Approaches to the Assessment of Achievement PDF
Preview Alternative Approaches to the Assessment of Achievement
ALTERNATIVE APPROACHES TO THE ASSESSMENT OF ACHIEVEMENT Evaluation in Education and Human Services Editors: George F. Madaus, Boston College, Chestnut Hill, Massachusetts, U.S.A. Daniel L. Stufflebeam, Western Michigan University, Kalamazoo, Michigan, U.S.A. Previously published books in the series: Kellaghan, T., Madaus, G., and Airasian, P.: The Effects of Standardized Testing Madaus, G. (editor): The Courts, Validity, and Minimum Competency Testing Brinkerhoff, R., Brethower, D., Hluchyj, T., and Nowakowski, J.: Program Evaluation, Sourcebook/ Casebook Brinkerhoff, R., Brethower, D., Hluchyj, T., and Nowakowski, J.: Program Evaluation, Sourcebook Brinkerhoff, R., Brethower, D., Hluchyj, T., and Nowakowski, J.: Program Evaluation, Design Manual Madaus, G., Scriven, M., Stufflebeam, D.,; Evaluation Models: Viewpoints on Educational and Human Services Evaluation Hambleton, R., Swaminathan, H. Item Response Theory Stufflebeam, D., Shinkfield, A.; Systematic Evaluation Nowakowski, J., Handbook of Educational Variables: A Guide to Evaluation Stufflebeam, D.; Conducting Educational Needs Assessments Abrahamson, Stephen; Evaluation of Continuing Education in the Health Professions Cooley, William, Bickel, William; Decision-oriented Educational Research Gable, Robert K.; Instrument Development in the Affective Domain Sirotnik, Kenneth A., Oakes, Jeannie; Critical Perspectives on the Organization and Improvement of Schooling Wick, John W.; School-based Evaluation: A Guide for Board Members, Superintendents, Principals, Department Heads and Teachers Worthen, Blaine R., White, Karl R.; Evaluating Educational and Social Programs ALTERNATIVE APPROACHES TO THE ASSESSMENT OF ACHIEVEMENT Edited by David L. McArthur UCLA Graduate School of Education Center for the Study of Evaluation ~ " Kluwer Academic Publishers Boston/ Dordrecht/ Lancaster DiSlribUIOU for NOrib Americl: Klu ..... er Aeademic Puhli~hers 101 Philip Drive Assinippi Park Norwell. MA 02061 USA DislribulOrs for Ibt UK 3nd Ireland: Kluwcr Academic Publishers MTP Press Limited Falcon House. Queen Square lancaster LAI IRN UNITED KINGDOM Oi~tribulo", for all olber countri", Kluwer Academic Publishers Group Distribution Cellire POSt Office BOJ! 322 3JOO A H Dord recht The Netherlands Library of C.mgrtss Cililoging-in-Publin.lion Oltl Ahernative approache~ to the assessment of achie~·emem. (Evaluation in education and human services) Includes bibliographies alld inde~. I. &locationall"sts and measurements. 2. Educational tcsts and measuremems- Mathcmatical models. J. Academic achie""menl- Tesling Mathematical models. I. McAnhur. David. II. Series. [O;\'lM: I Educational Measuremenl methods. 2. Evaluation Studies. LB 3051 A466] LB3051.A567 1986 )71.2'6 86-27483 tSIIN-!J: 1178·\1-1-011).7\161-7 <-tSBN-!J: \178-\1-1-009-3257-9 DOt: to.l007I97tt-9-I-009-J257-9 Copyright " 19S7 by Kluwer Academic I'ublishers Softco,"er reprint of the hardco,"er I st edit ion \987 All riShlS reserved. No part uf this publication may be reproduced. SlUred in a retrieval system. or transmiltcd ill an)' form or b)' an)' means. mechanical.llhotocopying. reo;ording. or otherwise. without the prioT wTiuen permission of the publisher. KIII",eT Academic Publishers. Philip Drive. Assinippi Park. Nor ....- cll. Massachu\ClIS 02061. USA. CONTENTS Contributors Vll Preface IX Introduction Xlll Bruce H. Choppin and David L. McArthur Chapter 1: Educational Assessment: A Brief History David L. McArthur Chapter 2: Toward More Sensible Achievement Measurement: 21 A Retrospective Kenneth A. Sirotnik Chapter 3: Analysis of Patterns: The S-P Technique 79 David L. McArthur Chapter 4: The Rasch Model for Item Analysis 99 Bruce H. Choppin Chapter 5: The Three-Parameter Logistic Model 129 Ronald K. Hambleton Chapter 6: Measuring Achievement with Latent Structure Models 159 Rand R. Wilcox Chapter 7: Generalizability Theory and Achievement Testing 187 Noreen M. Webb Chapter 8: Analysis of Reading Comprehension Data 233 David L. McArthur Chapter 9: A Comparison of Models for Measuring Achievement 249 J. Ward Keesling Index 267 LIST OF CONTRIBUTORS Bruce H. Choppin, PhD deceased; Dr. Choppin was Director of the Methodology Research program at the Center for Study of Evaluation Graduate School of Education University of California Los Angeles Ronald K. Hambleton, PhD Division of Educational Policy, Research and Administration University of Massachusetts at Amherst Amherst, MA 01003 J. Ward Keesling, PhD Computer Resources Special Project Advanced Technology, Inc. Camarillo, CA 93010 David L. McArthur, PhD Center for Student Testing, Evaluation and Standards University of California Los Angeles Graduate School of Education Los Angeles, CA 90024 Kenneth A. Sirotnik, PhD Policy, Governance and Administration College of Education University of Washington Seattle, WA 98195 Noreen M. Webb, PhD Research Methods in Education Graduate School of Education University of California Los Angeles Los Angeles, CA 90024 Rand R. Wilcox, PhD Quantitative Area Department of Psychology University of Southern California Los Angeles, CA 90089 PREFACE Ingrained for many years in the science of educational assessment were a large number of "truths" about how to make sense out of testing results, artful wisdoms that appear to have held away largely by force of habit alone. Practitioners and researchers only occasionally agreed about how tests should be designed, and were even further apart when they came to interpreting test responses by any means other than categorically "right" or "wrong." Even the best innovations were painfully slow to be incorporated into practice. The traditional approach to testing was developed to accomplish only two tasks: to provide ranking of students, or to select relatively small proportions of students for special treatment. In these tasks it was fairly effective, but it is increasingly seen as inadequate for the broader spectrum of issues that educational measurement is now called upon to address. Today the range of questions being asked of educational test data is itself growing by leaps and bounds. Fortunately, to meet this challenge we have available a wide panoply of resource tools for assessment which deserve serious attention. Many of them have exceptionally sOphisticated mathematical foundations, and succeed well where older and less versatile techniques fail dismally. Yet no single new tool can conceivably cover the entire arena. Our intent in this book is to convey to the reader a well-balanced appreciation for the diversity of alternative methodologies for educational assessment. To do this we have attempted to set a stage which portrays both a cross-section of methods and an intersection of the principles which underlie them. Included are evaluations of both the strengths and weaknesses of every approach, as well as completely worked-out examples which demonstrate their present utility, and informed speculations about future developments and applications to which they could be put. x Most mathematical models, at their heart, are quite simple; the statistical tricks are in place solely to make them work efficiently. From the reader who may be a trifle rusty in statistics, we ask only for patience if at times the various formulae for number--crunching make tough reading. Readers experienced in designing and interpreting educational tests may have an advantage on this score, though the book is also designed to be useful at introductory levels. Moreover, the essential ingredients of this book are not the formulae themselves but the competing rationales which precede them, and the variety of conclusions which follow. This book is the culmination of a project first set in motion by the late Bruce Choppin. Its overall design is his, and it was his imagination which first brought the contributors together to consider detailing alternative models for the assessment of educational achievement using a common framework. His desire to give full and fair scrutiny to contrasting viewpoints and his ability to draw connections across rival hypotheses with remarkable clarity, together with his wide command of theory and practice, were a strong influence on all who had the chance to work with him. One intent in this book has been to capture Bruce's original sense of marvel at how good questions can be raised--and answered well--in the field of educational assessment. Bruce suffered an untimely death while on the first leg of a consulting trip around the world. He continues to be missed by his many friends and colleagues. My heartfelt thanks go to each of the authors for their participation, and their willingness to develop each of the chapters in this book into documents which integrate their views along a common strand of critical issues. We hope that the reader will find this means of presenting the several alternative methods to be one which elucidates the essential philosophical bases beneath the measurement process. We have also strived to provide a consistent technical vocabulary necessary to convey statistical concepts, and frequent guideposts to applying the several models in actual practice. XI Support for work presented in this document was originally provided by the National Institute of Education, Grant Number NIE-G-80-112, P1. The opinions expressed herein do not necessarily represent the position or policy of the Institute, now known as the Office of Educational Research and Improvement, and no official endorsement should be inferred. Additional support was generously provided by the Center for Student Testing, Evaluation and Standards at UCLA. Special appreciation is extended to Katharine Fry, Ruth Pays en and Dan Reich for diligent typing of each contribution. David L. McArthur INTRODUCTION Fragmentation is occurring within the psychometric field. Dissatisfaction with the limitations inherent in traditional forms of mental test analysis typified by the norm-referenced multiple-choice test of achievement, has led in recent years to a variety of new psychometric theories and procedures. Novel applications have stimulated new psychometric models and methods, each shaped to deal with the specific problems of the particular situation. The last two decades have seen the development of new types of tests, new scoring methods, new procedures for item analysis, and entirely new conceptions of the mental measurement process. A marked characteristic of the professional literature on these novel approaches to educational measurement is its parochialism. Many of the most prolific psychometricians display little interest in models other than their own, and there have been few attempts to integrate separate results. The proponents of different models have different objectives which on first impression appear to be mutually exclusive. Implicitly or explicitly they make widely differing assumptions; frequently they use the same words and phrases to mean different things (e.g., reliability, accuracy, guessing, error and true-score). Separate methodologies based on different models have diverged to a point where it is no longer possible to identify a mainstream approach to educational measurement, and where informed and balanced advice on the full range of alternative approaches is almost impossible to obtain. The primary goal of this book is to build a convergent perspective on psychometrics by describing, fairly and comprehensively, the rationales that underlie several of the leading
The list of books you might like
