The Learning Sciences in Educational Assessment: The Role of Cognitive Models

283 Pages·2011·1.853 MB·English

by  Jacqueline P. Leighton & Mark J. Gierl

The Learning Sciences in Educational Assessment
Th ere is mounting hope in the United States that federal legislation in the
form of No Child Left Behind and the Race to the Top fund will improve
educational outcomes. As titanic as the challenge appears to be, however,
the solution could be at our fi ngertips. Th is volume identifi es visual types of
cognitive models in reading, science, and mathematics for researchers, test
developers, school administrators, policy makers, and teachers. In the process
of identifying these cognitive models, the book also explores methodological
or translation issues to consider as decisions are made about how to gen-
erate psychologically informative and psychometrically viable large-scale
assessments based on the learning sciences . Initiatives to overhaul educational
systems in disrepair may begin with national policies, but the success of these
policies will hinge on how well stakeholders begin to rethink what is possible
with a keystone of the educational system: large-scale assessment.
Jacqueline P. Leighton is Professor of Educational Psychology and Director
of the Centre for Research in Applied Measurement and Evaluation at the
University of Alberta. She is also registered as a psychologist by the College of
Alberta Psychologists. Her specialization is educational assessment and cog-
nitive psychology, with an emphasis on test development and validity analy-
sis. Dr. Leighton’s current research is on identifying and evaluating methods
for generating cognitive models for educational-assessment practice. Her
research has been funded by the Natural Sciences and Engineering Research
Council of Canada and the Canadian Education Statistics Council and is cur-
rently funded by the Social Sciences and Humanities Research Council of
Canada.
Mark J. Gierl is Professor of Educational Psychology at the University of
Alberta. His specialization is educational and psychological testing, with an
emphasis on the application of cognitive principles to assessment practices.
Dr. Gierl’s current research is focused on diff erential item and bundle func-
tioning, cognitive diagnostic assessment, and assessment engineering. His
research is funded by both the College Board and the Social Sciences and
Humanities Research Council of Canada. He holds the Canada Research
Chair in Educational Measurement.
Downloaded from Cambridge Books Online by IP 14.139.43.12 on Wed Oct 10 12:10:59 BST 2012.
http://ebooks.cambridge.org/ebook.jsf?bid=CBO9780511996276
Cambridge Books Online © Cambridge University Press, 2012
Downloaded from Cambridge Books Online by IP 14.139.43.12 on Wed Oct 10 12:10:59 BST 2012.
http://ebooks.cambridge.org/ebook.jsf?bid=CBO9780511996276
Cambridge Books Online © Cambridge University Press, 2012
Th e Learning Sciences in Educational
Assessment
The Role of Cognitive Models
Jacqueline P. Leighton
University of Alberta
Mark J. Gierl
University of Alberta

Downloaded from Cambridge Books Online by IP 14.139.43.12 on Wed Oct 10 12:10:59 BST 2012.
http://ebooks.cambridge.org/ebook.jsf?bid=CBO9780511996276
Cambridge Books Online © Cambridge University Press, 2012
cambridge university press
Cambridge, New York, Melbourne, Madrid, Cape Town,
Singapore, São Paulo, Delhi, Tokyo, Mexico City
Cambridge University Press
32 Avenue of the Americas, New York , ny 10013-2473, usa
www.cambridge.org
Information on this title: www.cambridge.org/9780521194112
© Jacqueline P. Leighton and Mark J. Gierl 2011
Th is publication is in copyright. Subject to statutory exception
and to the provisions of relevant collective licensing agreements,
no reproduction of any part may take place without the written
permission of Cambridge University Press.
First published 2011
Printed in the United States of America
A catalog record for this publication is available from the British Library.
Library of Congress Cataloging in Publication data
Leighton, Jacqueline P.
Th e learning sciences in educational assessment: the role of cognitive models/
Jacqueline P. Leighton, Mark J. Gierl.
p. cm.
Includes index.
isbn 978-0-521-19411-2 (hardback)
1. Educational tests and measurements. 2. Educational psychology.
3. Cognition. I. Gierl, Mark J. II. Title.
lb3051.l425 2011
371.260973–dc22 2011004251
isbn 978-0-521-19411-2 Hardback
Cambridge University Press has no responsibility for the persistence or accuracy of
urls for external or third-party Internet Web sites referred to in this publication and
does not guarantee that any content on such Web sites is, or will remain, accurate or
appropriate.

Downloaded from Cambridge Books Online by IP 14.139.43.12 on Wed Oct 10 12:10:59 BST 2012.
http://ebooks.cambridge.org/ebook.jsf?bid=CBO9780511996276
Cambridge Books Online © Cambridge University Press, 2012
Contents
1 Th e Learning Sciences in Educational Assessment:
An Introduction page 1
2 Evaluating Cognitive Models in Large-Scale Educational
Assessments 45
3 Cognitive Models of Task Performance for Reading
Comprehension 71
4 Cognitive Models of Task Performance for Scientifi c
Reasoning and Discovery 115
5 Cognitive Models of Task Performance for Mathematical
Reasoning 156
6 Putting It All Together: Cognitive Models to Inform
the Design and Development of Large-Scale Educational
Assessment 197
7 Cognitively Based Statistical Methods – Technical
Illustrations 234
Index 2 65
v
Downloaded from Cambridge Books Online by IP 14.139.43.12 on Wed Oct 10 12:11:01 BST 2012.
http://ebooks.cambridge.org/ebook.jsf?bid=CBO9780511996276
Cambridge Books Online © Cambridge University Press, 2012
Downloaded from Cambridge Books Online by IP 14.139.43.12 on Wed Oct 10 12:11:01 BST 2012.
http://ebooks.cambridge.org/ebook.jsf?bid=CBO9780511996276
Cambridge Books Online © Cambridge University Press, 2012
Cambridge Books Online
http://ebooks.cambridge.org/
The Learning Sciences in Educational Assessment
The Role of Cognitive Models
Jacqueline P. Leighton, Mark J. Gierl
Book DOI: http://dx.doi.org/10.1017/CBO9780511996276
Online ISBN: 9780511996276
Hardback ISBN: 9780521194112
Paperback ISBN: 9780521122887
Chapter
1 - The Learning Sciences in Educational Assessment: An Introduction p
p. 1-44
Chapter DOI: http://dx.doi.org/10.1017/CBO9780511996276.001
Cambridge University Press
1
Th e Learning Sciences in Educational
Assessment: An Introduction
V ictor Hugo is credited with stating that “Th ere is nothing more
p owerful than an idea whose time has come.” In educational achieve-
ment testing, 1 a multi-billion-dollar activity with profound implica-
tions for individuals, governments, and countries, the idea whose
time has come, it seems, is that l arge-scale achievement tests must be
designed according to the science of human learning . Why this idea,
and why now? To begin to set a context for this idea and this ques-
tion, a litany of research studies and public policy reports can be
cited to make the simple point that students in the United States and
abroad are performing relatively poorly in relation to expected stan-
dards and projected economic growth requirements (e.g., American
Association for the Advancement of Science, 1 993; Chen, Gorin,
Th ompson, & Tatsuoka, 2 008; Grigg, Lauko, & Brockway, 2 006;
Hanushek, 2003 , 2009 ; Kilpatrick, Swaff ord, & Findell, 2 001; Kirsch,
Braun, & Yamamoto, 2007 ; Manski & Wise, 1983 ; Murnane, Willet,
Dulhaldeborde, & Tyler, 2000 ; National Commission on Excellence
in Education, 1 983; National Mathematics Advisory Panel, 2 008;
National Research Council, 2005 , 2007 , 2009 ; Newcombe et al., 2009 ;
Phillips, 2007 ; Provasnik, Gonzales, & Miller, 2009 ). According to a
2007 article in the N ew York Times, Gary Phillips, chief scientist at the
American Institutes for Research , was quoted as saying, “our Asian
1 Th e terms “testing” and “assessment” are used interchangeably in the present
v olume to denote formal measurement techniques and evaluation procedures.
1
Downloaded from Cambridge Books Online by IP 14.139.43.12 on Wed Oct 10 12:11:05 BST 2012.
http://dx.doi.org/10.1017/CBO9780511996276.001
Cambridge Books Online © Cambridge University Press, 2012
2 Th e Learning Sciences in Educational Assessment
economic competitors are winning the race to prepare students in math
and science.” Phillips made this comment in relation to a report equat-
ing the standardized large-scale test scores of grade-eight students in
each of the fi ft y U.S. states with those of their peers in forty-fi ve coun-
tries. Underlying the sentiment in this quote is the supposition that test
scores reveal valuable information about the quality of student learn-
ing and achievement2 for feeding future innovation and economic
growth. If test scores reveal that U.S. students are underperforming
relative to their peers in other countries, then learning is likely being
compromised, and innovation and economic growth may also falter.
To change this potentially grim outcome, there are at least three
options: Change the educational system , change the large-scale tests,3
or change both. In the balance of this book, we focus on the sec-
ond option – changing the large-scale tests. Th is decision does not
indicate that the fi rst and third options lack merit. In fact, the third
option is ideal. However, in this fi rst chapter, we present a rationale
for why it makes sense to focus on changing tests, that is, to design
and develop large-scale educational assessments based on the learn-
ing sciences. To start, we discuss the relatively poor test performance
of many U.S. students as an impetus for the growing motivation in
North America to enhance the information large-scale educational
2 Although we recognize that learning and achievement sometimes connote diff er-
ent ideas (e.g., learning might be used in relation to the processes involved in the
acquisition of knowledge and skills, and achievement might be used in relation
to demonstrations of those knowledge and skills), learning and achievement are
used interchangeably in this volume. Th e goal of most educational initiatives and
institutions is to have learning and achievement overlap signifi cantly. In addition,
developers of achievement tests strive to design measures that are sensitive to
progressions in learning.
3 Th e focus is on large-scale educational testing because testing companies and
assessment branches of government agencies have the human and fi nancial cap-
ital to consistently develop, refi ne, and administrate standardized, psychometri-
cally sound assessments based on scientifi c cognitive learning models for large
numbers of students. Although classroom tests could also be developed from
fi ndings in the learning sciences, these are less likely to be developed according to
the same models due to a lack of resources.
Downloaded from Cambridge Books Online by IP 14.139.43.12 on Wed Oct 10 12:11:05 BST 2012.
http://dx.doi.org/10.1017/CBO9780511996276.001
Cambridge Books Online © Cambridge University Press, 2012
An Introduction 3
assessments currently provide about student learning. Next, we pres-
ent well-accepted knowledge from the learning sciences about the
nature of thinking, learning, and performance to help determine the
types of knowledge and skill components that may be required for
measurement as large-scale educational assessments are designed
and developed. Aft er that, illustrative empirical studies in the fi eld of
educational measurement are reviewed to demonstrate the nature of
the attempts to design and develop assessments based on the learning
sciences (also commonly referred to as cognitive diagnostic assessments
[CDA] or c ognitively based tests ; see Leighton & Gierl, 2007 ). Th en, we
off er a view on what is needed in the fi eld of educational assessment to
incorporate and systematically evaluate cognitive models in the design
and development of large-scale assessments. Finally, we present a con-
clusion and roadmap for the present volume that outlines the rationale
and content of the next six chapters, including what may be needed
to change large-scale tests and ensure they provide the information
about student learning and achievement many stakeholders seek.
The Impetus for Change: Low Achievement
Test Results
Th e U.S. Department of Education ( 2008 ) posted the following sum-
mary of the results achieved by fi ft een-year-old American students in
reading, science, and mathematics on the Programme for International
Student Assessment (PISA™) a dministered by the Organization for
Economic Cooperation and Development (OECD, 2007 , 2009 ; see
also U.S. results on Th ird International Mathematics and Science
Study [TIMSS] , Hanushek, 2009 ):
1. In the 2003 PISA administration , which focused on reading
literacy , U.S. students received an average score just higher
than the OECD average of approximately 500 (i.e., 495 versus
494, respectively; see OECD, 2 003) but lower than seventeen
Downloaded from Cambridge Books Online by IP 14.139.43.12 on Wed Oct 10 12:11:05 BST 2012.
http://dx.doi.org/10.1017/CBO9780511996276.001
Cambridge Books Online © Cambridge University Press, 2012