The think aloud method

219 Pages·2017·0.71 MB·English

UvA-DARE (Digital Academic Repository)
The think aloud method: a practical approach to modelling cognitive processes
van Someren, M.W.; Barnard, Y.F.; Sandberg, J.A.C.
Publication date
1994
Document Version
Author accepted manuscript
Link to publication
Citation for published version (APA):
van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994). The think aloud method: a
practical approach to modelling cognitive processes. (Knowledge-based systems). Academic
Press.
General rights
It is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s)
and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an open
content license (like Creative Commons).
Disclaimer/Complaints regulations
If you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please
let the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the material
inaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letter
to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. You
will be contacted as soon as possible.
UvA-DARE is a service provided by the library of the University of Amsterdam (https://dare.uva.nl)
Download date:12 Mar 2023
THE THINK ALOUD METHOD
A practical guide to modelling cognitive
processes
Maarten W. van Someren
Yvonne F. Barnard
Jacobijn A.C. Sandberg
Department of Social Science Informatics
University of Amsterdam
Published by Academic Press, London, 1994
ISBN 0-12-714270-3
Copyright M.W. van Someren, Y.F. Barnard and J.A.C. Barnard
Contents
1 Thinking aloud 1
1.1 A first impression . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Theories of cognitive processes in problem-solving . . . . . . . 8
1.3 Building knowledge-based systems . . . . . . . . . . . . . . . . 9
1.4 Overview of this book . . . . . . . . . . . . . . . . . . . . . . 12
2 Studying the content of cognitive processes 13
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2 Cognitive processes in problem-solving . . . . . . . . . . . . . 13
2.3 Observation methods . . . . . . . . . . . . . . . . . . . . . . . 15
2.4 Structured techniques . . . . . . . . . . . . . . . . . . . . . . . 16
2.5 Verbal reports . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.5.1 The verbal reporting process . . . . . . . . . . . . . . . 19
2.5.2 Retrospection . . . . . . . . . . . . . . . . . . . . . . . 21
2.5.3 Introspection . . . . . . . . . . . . . . . . . . . . . . . 23
2.5.4 Questions and prompting . . . . . . . . . . . . . . . . . 23
2.5.5 Dialogue observation . . . . . . . . . . . . . . . . . . . 24
2.6 Differences between methods . . . . . . . . . . . . . . . . . . . 24
2.7 Think aloud protocols . . . . . . . . . . . . . . . . . . . . . . 26
2.8 Combining methods . . . . . . . . . . . . . . . . . . . . . . . . 26
3 The think aloud method 29
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.2 History of the think aloud method . . . . . . . . . . . . . . . . 29
3.3 Selecting subjects . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.3.1 Criteria for selecting subjects . . . . . . . . . . . . . . 34
3.3.2 Experts as subjects . . . . . . . . . . . . . . . . . . . . 34
3.3.3 Differences in verbalization skills . . . . . . . . . . . . 35
3.4 Selecting problems . . . . . . . . . . . . . . . . . . . . . . . . 36
3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
vi The Think Aloud Method
3.6 Overview of the analysis of think aloud protocols . . . . . . . 37
4 Practical procedures in obtaining think aloud protocols 41
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
4.2 Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
4.3 Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.4 Warming up . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.5 Behaviour of the experimenter and prompting . . . . . . . . . 44
4.6 Recording . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.7 Transcription of the protocol . . . . . . . . . . . . . . . . . . . 44
4.8 Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
5 Building models of problem-solving 49
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
5.2 Modelling cognitive processes . . . . . . . . . . . . . . . . . . 49
5.3 The form of models of cognitive processes . . . . . . . . . . . . 51
5.4 Procedural models and explanation of human behaviour . . . . 53
5.5 Building models . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5.6 Task analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
5.6.1 The construction of a task analysis . . . . . . . . . . . 56
5.6.2 Example: task analysis of solving arithmetic word prob-
lems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5.6.3 Example: task analysis of architectural design . . . . . 63
5.6.4 The role of task analysis . . . . . . . . . . . . . . . . . 64
5.7 Theories of problem-solving . . . . . . . . . . . . . . . . . . . 65
5.7.1 The role of psychological theories . . . . . . . . . . . . 65
5.7.2 Example: psychological theories on solving arithmetic
word problems . . . . . . . . . . . . . . . . . . . . . . 68
5.7.3 Example: psychological theories on problem-solving in
architectural design . . . . . . . . . . . . . . . . . . . . 68
5.8 Psychological model . . . . . . . . . . . . . . . . . . . . . . . 69
5.8.1 The construction of psychological models . . . . . . . . 69
5.8.2 Example: a psychological model of solving arithmetic
word problems . . . . . . . . . . . . . . . . . . . . . . 70
5.8.3 Example: a psychological model of architectural design 73
5.9 Dimensions of models . . . . . . . . . . . . . . . . . . . . . . . 75
5.10 On the boundaries of task analysis and model construction . . 77
Contents vii
6 Languages for task analysis and psychological modelling 79
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
6.2 A conceptual modelling language . . . . . . . . . . . . . . . . 81
6.2.1 CPML (Conceptual Protocol Modelling Language) . . 81
6.2.2 Domain layer . . . . . . . . . . . . . . . . . . . . . . . 82
6.2.3 Inference layer . . . . . . . . . . . . . . . . . . . . . . . 84
6.2.4 Task layer . . . . . . . . . . . . . . . . . . . . . . . . . 86
6.2.5 Example: a CPML model of architectural design . . . . 91
6.2.6 Concluding remarks . . . . . . . . . . . . . . . . . . . . 95
6.3 Pseudo programming language . . . . . . . . . . . . . . . . . . 96
6.4 Problem Behaviour Graph and production rule systems . . . . 98
6.4.1 Problem Behaviour Graph (PBG) . . . . . . . . . . . . 98
6.4.2 Example: part of a PBG model of architectural design 99
6.4.3 Production rules . . . . . . . . . . . . . . . . . . . . . 101
6.4.4 Extensions of production rules . . . . . . . . . . . . . . 104
6.4.5 ProblemBehaviourGraphs,productionrulesystemsand
human memory . . . . . . . . . . . . . . . . . . . . . . 106
6.5 Programming languages . . . . . . . . . . . . . . . . . . . . . 110
6.6 Using a language or adapting it . . . . . . . . . . . . . . . . . 112
6.7 Differences between languages . . . . . . . . . . . . . . . . . . 113
7 Analysing the protocols 117
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
7.2 The role of protocols as data in research . . . . . . . . . . . . 119
7.3 Transcription and segmentation . . . . . . . . . . . . . . . . . 119
7.4 Coding scheme and verbalization theory . . . . . . . . . . . . 120
7.4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 120
7.4.2 Constructing a coding scheme . . . . . . . . . . . . . . 121
7.4.3 Grain size and aggregation . . . . . . . . . . . . . . . . 122
7.4.4 Special coding categories . . . . . . . . . . . . . . . . . 122
7.4.5 Coding form . . . . . . . . . . . . . . . . . . . . . . . . 123
7.4.6 Example: a coding scheme for architectural design . . . 123
7.4.7 Verbalization theory . . . . . . . . . . . . . . . . . . . 124
7.4.8 Example of a verbalization theory . . . . . . . . . . . . 125
7.4.9 Methodological requirements for the coding scheme . . 126
7.5 Coding procedures . . . . . . . . . . . . . . . . . . . . . . . . 127
7.5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 127
7.5.2 Aggregation . . . . . . . . . . . . . . . . . . . . . . . . 127
7.5.3 Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
7.5.4 Rating protocols or protocol fragments . . . . . . . . . 128
viii The Think Aloud Method
7.6 Intercoder reliability . . . . . . . . . . . . . . . . . . . . . . . 128
7.7 Comparing the coded protocols with the models . . . . . . . . 133
7.7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 133
7.7.2 Comparing protocols with procedural models . . . . . . 133
7.7.3 Issues in quantifying the fit . . . . . . . . . . . . . . . 135
7.7.4 Comparing sets of protocols . . . . . . . . . . . . . . . 136
7.8 Computer support tools for analysis . . . . . . . . . . . . . . . 136
7.8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 136
7.8.2 Indexing tools . . . . . . . . . . . . . . . . . . . . . . . 137
7.8.3 An implemented model as tool . . . . . . . . . . . . . . 137
7.9 Reporting the results of protocol analysis . . . . . . . . . . . . 137
8 Examples 141
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
8.2 Solving physics problems . . . . . . . . . . . . . . . . . . . . . 141
8.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 141
8.2.2 An example problem . . . . . . . . . . . . . . . . . . . 142
8.2.3 The model of advanced problem solving . . . . . . . . 142
8.2.4 Design of the experiments . . . . . . . . . . . . . . . . 145
8.2.5 A protocol . . . . . . . . . . . . . . . . . . . . . . . . . 146
8.2.6 Coding the protocol . . . . . . . . . . . . . . . . . . . 148
8.2.7 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
8.2.8 The sequence of tasks . . . . . . . . . . . . . . . . . . . 151
8.2.9 The completeness of the model . . . . . . . . . . . . . 151
8.2.10 The level of detail of the model . . . . . . . . . . . . . 152
8.2.11 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 152
8.3 Explaining novice errors in computer programming . . . . . . 152
8.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 152
8.3.2 The model . . . . . . . . . . . . . . . . . . . . . . . . . 152
8.3.3 Design of the study . . . . . . . . . . . . . . . . . . . . 154
8.3.4 An analysed protocol . . . . . . . . . . . . . . . . . . . 154
8.3.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 156
8.4 Acquisition of medical knowledge . . . . . . . . . . . . . . . . 157
8.4.1 Example: a medical diagnosis task . . . . . . . . . . . 157
8.4.2 Knowledge structures . . . . . . . . . . . . . . . . . . . 158
8.4.3 The problem-solving process . . . . . . . . . . . . . . . 160
8.4.4 Alternative models . . . . . . . . . . . . . . . . . . . . 163
8.4.5 Predicted and actual protocol . . . . . . . . . . . . . . 163
8.4.6 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
8.4.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 167
Contents ix
A Exercises 169
A.1 Exercise 1: collecting verbal data . . . . . . . . . . . . . . . . 169
A.2 Exercise 2: applicability of the think aloud method . . . . . . 170
A.3 Exercise 3: task analysis and model construction . . . . . . . . 170
A.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 170
A.3.2 Example problem . . . . . . . . . . . . . . . . . . . . . 171
A.3.3 Suggestions for task analysis and psychological model . 172
A.3.4 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . 174
A.4 Exercise 4: knowledge acquisition . . . . . . . . . . . . . . . . 174
A.5 Exercise 5: physics problem solving . . . . . . . . . . . . . . . 175
B Instructions for two problem-solving tasks 177
B.1 Task 1: waterjug problems . . . . . . . . . . . . . . . . . . . . 177
B.2 Task 2: improving technical devices . . . . . . . . . . . . . . . 177
C Protocols of ‘learning word meanings’ 179
D Analysing expert problem-solving 187
D.1 Case description . . . . . . . . . . . . . . . . . . . . . . . . . . 187
D.2 Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
E Coding scheme architectural design 195
F Protocol of novice problem-solving in physics 201
Bibliography 205
x The Think Aloud Method
Preface
This book gives a detailed description of the think aloud method. The think
aloud method consists of asking people to think aloud while solving a problem
and analysing the resulting verbal protocols. This method has applications in
psychological and educational research on cognitive processes but also for the
knowledge acquisition in the context of building knowledge-based computer
systems. In many cases the think aloud method is a unique source of infor-
mation on cognitive processes. In this book we present the method in detail
with examples.
Thisbookisintendedfortwotypesofreaders: socialscientistswhowantto
use the think aloud method for research on cognitive processes and knowledge
engineers who want to use the method for knowledge acquisition. To make the
book readable for both audiences, it contains short introductions to issues that
are basic knowledge for one readership, but that are not part of the standard
knowledge in the other community. We have included introductory sections
on those topics that are relevant for both communities. As a result the book
presupposes almost no specific knowledge but it is written for readers who
are basically familiar with one of the two major application areas of the think
aloudmethod: researchoncognitiveprocessesandknowledgeacquisition. The
role of the think aloud method (and related techniques) is explained separately
for psychological research and for knowledge acquisition. The book discusses
the aspects of computer models that are directly associated with the think
aloud method but not programming techniques.
This book has grown out of a long tradition at the University of Amster-
dam. The tradition started in the 1930s with Otto Selz who used the think
aloud method to study the creative reasoning processes. In the 1940s A.D.
de Groot used the method in his famous study of thought processes in chess.
In the 1960s and 1970s Jan Elshout and his colleagues used the method in
detailed process studies of cognitive skills that were related to general intelli-
gence. In this period Elshout also designed the first university course in which
the method was taught. Many of the ideas presented in this book originate
xii The Think Aloud Method
from this work. Bob Wielinga introduced techniques for computational mod-
elling into the course and the method. This book is based directly on material
used in a course on the think aloud method as it is taught at the University of
Amsterdam. We hope to provide a practical guide to all who like to use the
method in research or teach the method in a course.
Acknowledgments
The authors are grateful to all students who commented on earlier drafts of
this text. Furthermore we like to thank Ronald Hamel and Wouter Janswei-
jer who allowed us to use their work on architectural design and on physics
problem-solving as illustrative material in this book and who gave many useful
commentsonearlierdrafts. FinallywethanktheDepartmentofSocialScience
Informatics at the Faculty of Psychology of the University of Amsterdam for
the time and resources that allowed us to write this book.
The protocol texts of architects in Chapter 1, Figure 5.2 and the cod-
ing scheme in Appendix E were adapted with permission of R. Hamel: Het
denken van de architect, AHA Books, 1990, distributed by Staatsdrukkerij en
Uitgeverij (SDU), ’s Gravenhage.
The protocols of physics problem solving and the description of the model
of physics problem solving in Chapter 8 were adapted with permission of
W.H.N. Jansweijer: PDP: Een benadering vanuit de kunstmatige intelligentie
van probleemoplossen en leren door doen in een semantisch rijk domein, 1988,
PhD thesis, University of Amsterdam.