Assessment in Health Professions Education

346 Pages·2018·7.91 MB·English

by  Rachel Yudkowsky (editor), Yoon Soo Park (editor), Steven M. Downing (editor)

Assessment in Health Professions Education
Assessment in Health Professions Education, second edition, provides a comprehensive guide for
educators in the health professions—medicine, dentistry, nursing, pharmacy, and allied health
fields. This second edition has been extensively revised and updated by leaders in the field.
Part I of the book presents an introduction to assessment fundamentals and their theoretical
underpinnings from the perspective of the health professions. Part II covers specific assessment
methods, with a focus on validity, best practices, challenges, and practical guidelines for the
effective implementation of successful assessment programs. Part III addresses special topics
and recent innovative approaches, including narrative assessment, situational judgment tests,
programmatic assessment, mastery learning settings, and the key features approach. This
accessible text addresses the essential concepts for the health professions educator and provides
the background needed to understand, interpret, develop, and effectively implement assessment
methods.
Rachel Yudkowsky is Professor and Director of Graduate Studies in the Department of Medical
Education in the College of Medicine at the University of Illinois at Chicago, USA, and past
Director of the UIC Dr. Allan L. and Mary L. Graham Clinical Performance Center.
Yoon Soo Park is Associate Professor and Associate Head of the Department of Medical
Education, and Director of Research for Educational Affairs in the College of Medicine at the
University of Illinois at Chicago, USA.
Steven M. Downing is Associate Professor, Emeritus, in the Department of Medical Education in
the College of Medicine at the University of Illinois at Chicago, USA.

Assessment in
Health Professions
Education
Second edition
Edited by
Rachel Yudkowsky, Yoon Soo Park,
and Steven M. Downing
Second edition published 2020
by Routledge
52 Vanderbilt Avenue, New York, NY 10017
and by Routledge
2 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN
Routledge is an imprint of the Taylor & Francis Group, an informa business
© 2020 Taylor & Francis
The right of Rachel Yudkowsky, Yoon Soo Park, and Steven M. Downing to be identified as the
authors of the editorial material, and of the authors for their individual chapters, has been asserted
in accordance with sections 77 and 78 of the Copyright, Designs and Patents Act 1988.
All rights reserved. No part of this book may be reprinted or reproduced or utilised in any form
or by any electronic, mechanical, or other means, now known or hereafter invented, including
photocopying and recording, or in any information storage or retrieval system, without permission
in writing from the publishers.
Trademark notice: Product or corporate names may be trademarks or registered trademarks,
and are used only for identification and explanation without intent to infringe.
First edition published by Routledge 2009
Library of Congress Cataloging-in-Publication Data
A catalog record for this book has been requested
ISBN: 978-1-138-05438-7 (hbk)
ISBN: 978-1-315-16690-2 (pbk)
ISBN: 978-1-138-05439-4 (ebk)
Typeset in Minion
by Apex CoVantage, LLC
To my husband Moshe
1.3 billion seconds and counting
To my parents and teachers, Moshe and Mimi Soller
Whose love and values are my bedrock
To our children Eliezer, Channah and Jeff
and our grandchildren Aryeh and Shifra
Who bring us so much pride and joy
And in memory of our son Yehuda Nattan
May his memory be a blessing to all who loved him.
—RY
To my wife Heeyoung
For your love, patience, and encouragement
To my parents Sanghyun Park and Hyeyoung Jin
Whose prayers and love shape my decisions
To my teachers and colleagues
Who inspire me to continue the work we all enjoy.
—YSP

CONTENTS
List of Figures xvii
Foreword xix
Georges Bordage
Preface xxi
PART I FUNDAMENTALS OF ASSESSMENT 1
Chapter 1 Introduction to Assessment in the Health Professions 3
Rachel Yudkowsky, Yoon Soo Park, and Steven M. Downing
George Miller’s Pyramid 4
Four Major Assessment Methods 5
Written Tests 5
Oral Examinations 6
Performance Tests 6
Workplace-Based Assessment (Clinical Observational Methods) 7
Narrative Assessments and Portfolios 7
Some Basic Terms and Definitions 7
Competency-Based Education 7
Instruction and Assessment 9
Assessment, Measurement, and Tests 10
Types of Numbers 10
Fidelity to the Criterion 11
Formative and Summative Assessment 12
vii
viii • Contents
Norm- and Criterion-Referenced Measurement 12
High-Stakes and Low-Stakes Assessments 13
Large-Scale and Local or Small-Scale Assessments 13
Translational Science in Education 14
Summary 15
References 15
Chapter 2 Validity and Quality 17
Matthew Lineberry
The Principle of Purpose-Driven Assessment 17
Understanding Your Purposes 18
Tying Methods to Your Purposes 19
Investigating Validity, Evaluating Quality 20
Kane’s Validity Framework: The Validity Argument 21
Messick’s Validity Framework: Sources of Evidence 24
Evaluation Frameworks 27
Taking a Prevention Focus: Understanding Threats to Validity 27
Concluding Thoughts 32
References 32
Chapter 3 Reliability 33
Yoon Soo Park
Introduction 33
Theoretical Framework for Reliability 34
An Analogy 34
Classical Test Theory (CTT) 34
Illustrative Example 35
Implications 35
Reliability Indices: Test-Based Methods 35
1. Test-Retest Reliability 36
2. Split-Half Reliability 38
3. Internal-Consistency Reliability: Cronbach’s Alpha 39
Reliability Indices: Raters 41
Conceptual Basis for Inter-Rater Reliability 41
Illustrative Example: Inter-Rater Reliability 41
Implications: Inter-Rater Reliability 41
Standard Error of Measurement (SEM) 42
Illustrative Example 42
Implications 43
How to Increase Reliability 43
Projections in Reliability: Spearman-Brown Formula 43
Illustrative Example 44
Implications 44
Composite Scores and Composite Score Reliability 44
Illustrative Example 44
Implications 45
Contents • ix
Summary 45
Appendix: Supplement—Composite Scores and Composite Score
Reliability 47
Objective 47
What We Know 47
I. Calculating the Composite Score Reliability 48
II. Calculating the Composite Scores of the Three
Exemplar Students 49
References 50
Chapter 4 Generalizability Theory 51
Clarence D. Kreiter, Nikki L. Zaidi, and Yoon Soo Park
Introductory Comments 51
Background and Overview 51
The Hypothetical Measurement Problem—An Example 53
Defining the G Study Model 53
Obtaining G Study Results 54
Interpreting G Study Results 55
Conducting the D Study 56
Interpreting the D Study 58
G and D Study Model Variations 59
Unbalanced Designs 60
Multivariate Generalizability 61
Additional Considerations 63
Final Considerations 64
Appendix 4.1: Statistical Foundations of a Generalizability Study 65
Appendix 4.2: Statistical Foundations of a Decision Study 67
Appendix 4.3: Software Syntax for Estimating Variance
Components From Table 4.1 Data 68
SPSS Syntax for Estimating Variance Components From
Table 4.1 Data 68
SAS Syntax for Estimating Variance Components From
Table 4.1 Data 68
GENOVA Syntax for Estimating Variance Components From
Table 4.1 Data 68
References 69
Chapter 5 Statistics of Testing 70
Steven M. Downing, Dorthea Juul, and Yoon Soo Park
Introduction 70
Using Test Scores 70
Basic Score Types 70
Number Correct Scores or Raw Scores 71
Percent-Correct Scores 71
Derived Scores or Standard Scores 72
Normalized Standard Scores 73