Analytical reasoning with multiple external representations

268 Pages·2002·1.57 MB·English

Analytical reasoning with multiple external representations
Richard Cox
U N I VER
E S
I
H
T
T Y
O H
G
F
R
E U
DI N B
Ph.D.
University of Edinburgh
1996
Abstract
This thesis presents work on analytical reasoning with external representations (ERs)
using problems similar to those used in the US GRE college-entrance examination.
The work investigates the factors associated with e(cid:11)ective ER use in situations where
subjectsselect, constructand reasonwiththeir ownERs. Practically all previous work
has tended to focus solely upon performance rather than process. In this thesis the
emphasis is upon cognitive processes during the entire time-course of reasoning with
ERs, from problem comprehension through to answer selection. A background to the
work is provided by 2 comprehensive reviews of: 1.) previous research on ERs and
reasoning and 2.) the cognitive and semantic properties of ERs.
Results from three empirical studies are reported. The (cid:12)rst study examined a large
corpusof‘workscratchings’produced bysubjectsastheysolvedpaperandpencil-based
analytical reasoning problems under test conditions. The workscratching ERs showed
great diversity between and within subjects and across a range of problems. They
included lists, various kinds of table, set diagrams, node and arc diagrams, (cid:12)rst-order
and propositional logic, plans and natural language. It is shown that problem-solving
performance is related to the type of ER used in the solution.
The second study utilised a computer-based system (switchERI). The system admin-
istered analytical reasoning problems and provided a range of ER construction envi-
ronments for the subject to choose and switch between. User-system interactions were
recorded dynamically during problem solving. This methodology permitted micro-
analyses of the cognitive events at each stage during the time-course of problem solv-
ing. A process account of analytical reasoning with ERs is developed in which (cid:12)ve
major stages are identi(cid:12)ed - problem comprehension, ER selection, ER construction,
read-o(cid:11) from the ER and answer selection/responding. A range of common slips and
misconceptions are identi(cid:12)ed at each stage. The results show, inter alia, that subjects
whose responses are consistent with their ERs perform better than subjects whose
responses are inconsistent with their ERs even if the ER is partially incorrect.
The data from the workscratching analysis and switchERI study informed the design
of switchERII, a second system. SwitchERII incorporates a representation of the
semantics of Euler’s Circles, dynamically parses the user’s representation and provides
feedback and advice. A third study was conducted with the switchERII system.
Few, if any, studies to date have attempted to relate subjects’ prior knowledge of ER
formalismstotheirreasoningperformance. Subjects’priorknowledgeofERformalisms
was assessed in both switchER studies. It was observed that subjects’ performance on
representation interpretation tasks does not necessarily predict their performance in
conditions where they select and construct their own representations. The reasons for
the decoupling are discussed.
Data from all three studies show that subjects often utilise multiple representations
in their solutions, either concurrently or serially via ER switching. Two distinctly
di(cid:11)erent types of switching were observed. One kind (‘thrashing’) is associated with
poorer performance and re(cid:13)ects less comprehensive prior knowledge, inability to select
anappropriateERandhazyproblemcomprehension. Judicious switching,ontheother
hand, is associated with high levels of problem comprehension and skilled matching of
ii
the ERs’ properties to changing task demands.
It is claimed that e(cid:11)ective reasoning with ERs involves complex interactions between
at least three factors: (a.) within-subject variables such as the subject’s representa-
tionalrepertoire(priorknowledge)andrepresentationalmodalitypreferences(cognitive
style); (b.) skill at overcoming a variety of barriers to comprehension and an ability to
discern the salient attributes and characteristics of di(cid:11)erent problem types and (c.) an
understanding of the semantic and cognitive properties of graphical and non-graphical
ERs coupled with an ability to match those properties to the problem’s task demands.
It is suggested that the role of externalisation in reasoning with ERs may be to fa-
cilitate the swapping of information between cognitive subsystems. A mechanism by
which the use of diagrammatic ERs may facilitate self-explanation is also proposed.
Thethesis concludes with anargumentin favourofadomain-independent ‘ERcurricu-
lum’ . It is suggested that direct instruction in the use of a range of ERs might equip
students with wider representational repertoires and hence allow them more scope to
indulge their representational preferences. Finally, several directions for future work
are proposed. These include extending the representational semantics of switchERII,
evaluating various types of system feedback and implementing a mechanism for check-
ing for slips during read-o(cid:11) from ERs.
iii
Acknowledgements
This thesis is dedicated to my family.
Many individuals are owed my thanks and gratitude for helping bring this thesis to its
conclusion.
First, profound and special thanks are due to my primary supervisor, Dr. Paul Brna,
for many hours of patient help and supervision and for being a constant source of
advice and encouragement.
Secondly, IwouldliketothankmyworkcolleaguesintheGraphicsandLanguageWork-
ing Group of the Human Communication Research Centre - Professor Keith Stenning
(my second supervisor), Drs. Corin Gurr, Robert Inder, John Lee, Jon Oberlander,
andRichardTobinforinnumerable stimulatingdiscussions, somein formalsettingsbut
mostly in our o(cid:14)ces and in the tea room! I also acknowledge the invaluable assistance
A
that Robert and Richard gave me with Prolog and LTEX.
Thanks, for many stimulating discussions in Cellar One (and more formal settings),
to my friends and colleagues in the A.I. and Education and COED groups, especially
Maziah Aziz, Diana Bental, Susan Bull, David Duncan and Judith Good. I also wish
to thank everyone who participated in my experiments.
I must also express gratitude to Dr. Helen Pain and Professor Rosamund Sutherland
for agreeing to formally examine the thesis.
I would also like to acknowledge the continuing in(cid:13)uence of my Masters degree super-
visor, Dr. Tom Matyas (School of Behavioural Health Sciences, La Trobe University,
Australia). I remain indebted to him for any skills in research methods and statistics
that I may possess. Thanks too to Dr. Geo(cid:11) Cumming of the Psychology Department
for many useful and thought-provoking discussions.
Finally, I am indebted to Dr. Carmel Lum, my dear friend and partner of 19 years,
whose love and constant encouragement enabled me to realise my aspirations.
iv
Declaration
I hereby declare that I composed this thesis entirely myself and that it describes my
own research.
Richard Cox
Edinburgh
December 4, 1996
Some of the work in this thesis was published prior to thesis submission. Some of
the analyses of ‘workscratchings’ presented in Chapter 5 and the switchERI study
described in Chapter 6 appeared as:
Cox, R. & Brna, P. [1995] Supporting the use of external representations in prob-
lem solving: The need for (cid:13)exible learning environments. Journal of Arti(cid:12)cial
Intelligence in Education, 6(2), 239{302.
Some of the workscratching analyses reported in Chapter 5 were also reported in:
Cox, R., Stenning, K. & Oberlander, J. [1995] The e(cid:11)ect of graphical and sen-
tential logic teaching on spontaneous external representation. Cognitive Studies:
Bulletin of the Japanese Cognitive Science Society, 2(4), 56{75.
Several conference papers were also based on the work:
Cox, R.J. & Brna, P. [1993] Reasoning with external representations: Supporting
the stages of selection, construction and use. Proceedings of the World Con-
ference on Arti(cid:12)cial Intelligence in Education (AI-ED93), Charlottesville, VA:
Association for the Advancement of Computing in Education, 185-192.
Cox, R.J. & Brna, P. [1993] Analytical reasoning with external representations:
The relationship between prior knowledge and performance. In R.Cox, Petre,
M., Brna, P. and Lee, J. Proceedings of the workshop on Graphical represen-
tations, reasoning and communication held at World Conference on Arti(cid:12)cial
Intelligence and Education (AI-ED93), August, Edinburgh, 33-36.
Cox, R. & Draper, S. [1995] External representations, choice and task e(cid:11)ective-
ness. 4th ESRC Seminar on internal and external representations (Representa-
tions and Constructivism), Institute of Education, London, 21-22 April, 1995.
v
Cox, R. [1996] The role of externalisation in reasoning with self-constructed repre-
sentations. Presented at the IEE Colloquium on Thinking with Diagrams,Savoy
Place, London, January 18. IEE Digest No. 96/010, London: Institute of Elec-
trical Engineers, 1{7.
The following Departmental papers were also based on the work:
Cox, R.J. & Brna, P. [1993] The relationship between prior knowledge of external
representationsandanalyticalreasoningperformance: Implications forthedesign
of a learning environment. Research Paper RP-646, Department of Arti(cid:12)cial
Intelligence, University of Edinburgh.
Cox, R. & Brna, P. [1994] Analyticalreasoningwithexternalrepresentations: Sup-
porting the stages of selection, construction and use. Research Paper RP-686,
Department of Arti(cid:12)cial Intelligence, University of Edinburgh.
vi
Contents
Abstract iii
Acknowledgements iv
Declaration vi
List of Figures viii
1 Introduction 1
What this thesis is about . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
What this thesis is not about . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
De(cid:12)ning the term ‘external representation’ . . . . . . . . . . . . . . . . . . . 5
ER taxonomies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Conceptualisations of representations in related (cid:12)elds . . . . . . . . . . . . . . 11
Representations in mathematics and arti(cid:12)cial intelligence . . . . . . . . 11
Representations and psychology, cognitive science . . . . . . . . . . . . . 15
Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2 The domain: Analytical reasoning problems 25
The characteristics of analytical reasoning problems . . . . . . . . . . . . . . 25
Attractiveness of the domain . . . . . . . . . . . . . . . . . . . . . . . . 28
The linguistic and structural properties of analytical reasoning problems. 29
3 Stages in analytical reasoning with ERs 32
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Interpretation and comprehension . . . . . . . . . . . . . . . . . . . . . . 33
vii
ER selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
ER construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
Read-o(cid:11) from ER and responding . . . . . . . . . . . . . . . . . . . . . . 58
Summary and conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4 Related work 66
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
ER construction by the subject: studies utilising domain-speci(cid:12)c ER
forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
ER construction by the subject: studies utilising non domain-speci(cid:12)c
ER forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Studies in which subjects used prefabricated ERs . . . . . . . . . . . . . 79
Review conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
ERs and ILEs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Thesis arguments expounded in detail . . . . . . . . . . . . . . . . . . . . . . 88
ER selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Constructivism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Switching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Summary of aims of (cid:12)rst study . . . . . . . . . . . . . . . . . . . . . . . 93
5 An investigation of ER use in reasoning. 95
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Issues of validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Problem selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Subjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
Results and Discussion | Workscratching Corpus . . . . . . . . . . . . 99
Multiple representations . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Summary and Conclusions | Workscratching Data. . . . . . . . . . . . 109
6 SwitchERI - an ILE for analytical reasoning. 112
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
viii
Prior knowledge. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Method and Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
Subjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
Task Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Summary and Conclusions | switchERI study . . . . . . . . . . . . . . . . . 134
Implications for the Design of an Interactive Learning Environment (ILE)135
Support at the Problem Comprehension Stage. . . . . . . . . . . . . . . 136
Support at the Representation Selection Stage. . . . . . . . . . . . . . . 136
Supporting ER Construction . . . . . . . . . . . . . . . . . . . . . . . . 139
Supporting the Use of the ER in Question Answering . . . . . . . . . . . 142
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
7 SwitchERII { an intelligent ILE. 145
switchERII development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
Pilot study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
Subjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
Discussion and conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . 184
8 General thesis discussion 187
Summary of what has been achieved . . . . . . . . . . . . . . . . . . . . . . . 187
Problem comprehension . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
ER selection & construction . . . . . . . . . . . . . . . . . . . . . . . . . 190
Constructivism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
Individual di(cid:11)erences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
The way forward - future work . . . . . . . . . . . . . . . . . . . . . . . . . . 198
Graphics curriculum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
9 Conclusion 204
Thesis contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
ix
Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
10 References 209
A 227
Three examples of analytical reasoning problems . . . . . . . . . . . . . . . . 227
Example ERs - workscratching (cid:12)gures . . . . . . . . . . . . . . . . . . . . . . 230
Single Models | Ignoring Alternatives . . . . . . . . . . . . . . . . . . . . . . 246
‘Invented’ Annotations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
The Use of Multiple Representations . . . . . . . . . . . . . . . . . . . . . . . 248
Representation recommended by Brownstein et al. for Problem 2 . . . . . . . 250
Eighty-seven stimulus items used in the switchERI study taxonomy pre-task 251
Euler’s circle interpretation task data from switchERII study. . . . . . . . . . 254
B 255
Publications bound into thesis . . . . . . . . . . . . . . . . . . . . . . . . . . 255
x