The Thinking Aloud method in the evaluation workshop of the exhibition Ötzi

17 Pages·2011·1.34 MB·English

How to access the experiences
of museum visitors?

The Thinking Aloud method in the evaluation workshop

of the exhibition Ötzi – Iceman of the Alps

Tomi Heikkilä
Carina Jaatinen
Eeva Kyllönen
Leena Tornberg
2011
1. Introduction .................................................................................. - 3 -

2. Starting points of the workshop .................................................... - 3 -

3. The Thinking Aloud method .......................................................... - 3 -

3.1. Origins of the method .................................................................. - 3 -

3.2. Use of the Thinking Aloud method ............................................... - 4 -

3.3. How to “Think Aloud” in research? .............................................. - 4 -

4. Workshop implementation............................................................ - 5 -

4.1. Planning stage .............................................................................. - 5 -

4.2. In the workshop ............................................................................ - 7 -

4.3. Immediate observations and feedback discussion in the Exhibition
Café ..................................................................................................... - 8 -

4.4. Subsequent processing and analysis of evaluation material ..... - 10 -

4.6. Recorders’ interviews ................................................................. - 14 -

4.7. Did the workshop meet its objectives? ....................................... - 14 -

5. The Thinking Aloud method in museums .................................... - 15 -

7. Team behind the report .............................................................. - 16 -

References on the topic .................................................................. - 17 -
1. Introduction Consultant Beverly Serrell. Her model allows museum professionals to
evaluate the exhibition themselves. However, this time aim was to find an

approach that would also allow the everyday customer to be heard. The
The Finnish Museums Association organised the Exhibition Café -training
Thinking Aloud method was selected on Leena Tornberg’s suggestion.
event at WeeGee Exhibition Centre in Espoo on September 28–29, 2010.

During the brainstorming stage, the purpose and aims of the workshop
The training event included an evaluation workshop where the
were defined as follows:
participants could evaluate the exhibition Ötzi – Iceman of the Alps in the

KAMU exhibition facility of the Espoo City Museum. The exhibition
 Collection of the experiences of exhibition visitors to develop
comprised the international travelling exhibition, produced by the South
exhibition planning
Tyrol Museum of Archaeology, and the local Life here in Ötzi’s time
 Accessing the perspectives of different user groups
section, produced by the Espoo City Museum. The workshop was carried
 Testing the Thinking Aloud method to access visitor experiences
out by applying the Thinking/Talking Aloud method.
 Enhancing the awareness and mutual understanding of the entire

museum staff in order to develop more customer-oriented
Workshop planning was conducted by the Espoo City Museum’s
exhibition activities
exhibition and museum pedagogy units in cooperation with Project

Coordinator Leena Tornberg of the project Open Museum – a developing
learning environment for adults in the CICERO Learning Network of the
3. The Thinking Aloud method
University of Helsinki.

The team wishes to thank all those who took part in the workshop and The Thinking Aloud method aims to access the thoughts and experiences
hopes that this report will encourage museums to boldly try out different of the participant when he/she is behaving in a research situation.
methods to access visitor experiences.
3.1. Origins of the method

2. Starting points of the workshop
The Thinking Aloud method was developed in the field of usability and
consumer research as well as problem-solving. It is used most frequently
The starting point of the workshop was the Espoo City Museum’s interest in cognitive science and cognitive learning research. The method is also
in developing tools for customer-oriented exhibition planning. Customer- known as protocol analysis, because the material is usually transcribed
orientation is based on information on the customer’s needs and afterwards.
experiences. But how can this information be accessed? The Exhibition
Café provided an opportunity for an open-minded experiment. The inspiration for developing the method sprung from methodological
problems that were observed in various questionnaire studies.
Exhibition evaluation was also tried out in the very first Exhibition Café, Researchers discovered that in e.g. product trials, test users who filled in
organised in Tampere in 2007. The experiment was conducted using the questionnaires and answered interview questions afterwards often
Judging Exhibitions model developed by the American Museum Exhibition
- 3 -
experienced difficulties in remembering what their actual thoughts were 1. The participant is asked to verbalise all of his/her thoughts while
when using the product. In addition, it was observed that test users may he/she is performing a set task. This level aims to access thoughts
try to provide the answers that they expect the questioner to anticipate. that are not modified before they are put into words. The
Participants may also feel the need to appear more witty and intelligent participant is asked to speak either to him/herself, to another
than they were in the research situation. For example, although the participant (group) or to the researcher who is present. This level
researcher may have observed that solving something appeared difficult, of the method was used in the evaluation workshop of the
participants may afterwards declare that it was easy. Exhibition Café.
2. The participant is asked to explain or interpret his/her thoughts
In the 1980s, Clayton Lewis conducted the first studies on the according to the instructions provided by the researcher. At this
development of computer usability for IBM. However, the real developers level, the participant modifies his/her mental images or abstract
of the method were researchers Ericsson and Simon in the 1980s–1990s. concepts before putting them into words.
The method has also been used in self-observation in psychology, when 3. The participant tries to analyse his/her thinking. He/she justifies
the participant wishes or tries to study e.g. his/her functional processes. e.g. personal choices out loud or ponders and analyses the
In Finland, Pirita Seitamaa-Hakkarainen, among others, has actively used reasons behind them. This level refers to the participant’s process
the method in her research on the design of handicraft science. thinking.

The Thinking Aloud method is considered to be highly suitable for
3.2. Use of the Thinking Aloud method
studying information stored in the short-term memory. For instance, after

an exhibition visit, customers quickly forget parts of what they really
The Thinking Aloud method involves asking participants to verbalise their
thought and experienced in the exhibition. This is why the Thinking Aloud
thoughts while they are acting in a research situation. Participants are
method is especially suitable for visitor research conducted during an
allowed to speak in peace so that the observer does not interrupt their
exhibition visit.
train of thought. Participants should ideally speak non-stop, because
people also think constantly. Compared to a study based on an interview
3.3. How to “Think Aloud” in research?
and questionnaire, the benefit of the method is that participants do not
have the time to interpret or explain their thoughts.
The Thinking Aloud method could be compared to the internal speech
As in other data collection methods, the risk is that participants knowingly that we sometimes use when solving a difficult problem or pondering a
or accidentally leave some thoughts unexpressed or they do not vocalise thought by ourselves. Basically, all the advice that the participants need
their thoughts truthfully. It is possible to try to prevent these problems by is: think your thoughts out loud. This may be more difficult in the research
selecting participants carefully and providing clear instructions and situation than one might expect. Reference literature advises researchers
guidance. Participation in the study is usually voluntary, so it is unlikely to practice the method with participants in advance. Practicing is even
that there is any motivation to lie on purpose. more important if the material is collected using a technical device
(separate microphone, dictating machine, iPod) or if the material is
recorded on film. During the practice, both the participant and the
Ericsson and Simon (1996) distinguish three levels for when a person is
researcher can interrupt the situation and address any uncertainties,
asked to verbalise his/her thoughts.
- 4 -
because this should be avoided in the actual situation. The researcher proposal opened up a good opportunity also for practical cooperation.
may ask participants to speak either by focusing on a perspective or topic The decision was made to try out the Thinking Aloud method of which
assigned in advance or by free association, in which case the researcher none of the members of the team had previous practical experience.
personally selects the thoughts from the recorded speech that he/she is
interested in during the analysis. The Thinking Aloud method can be used Actual workshop planning began in the beginning of September. Three
with anyone who is able to verbalise their thoughts. preparative meetings were held during September (about 3h/meeting +
preparations + follow-up work).

4. Workshop implementation The purpose of the first meeting was to discuss the objectives and
implementation methods of the workshop. The kind of information that

could be collected from visitors was also considered. Proposals included
Planning of the workshop began in August 2010 and it was held during the
information on structures, technology, manners of presentation, factual
Exhibition Café in September. The material was analysed during October-
content, moving in the facility and the feelings evoked by the exhibition.
December, and the report was completed in March 2011. The study was
Initial plans also included pre-post questionnaires that would examine the
conducted alongside other work and lasted a total of seven months from
change of the visitor experience. Since it was not possible to use
the first idea meeting to the publication of the report.
recording devices in the workshop, possible ways to document the

visitors’ speech were also considered. When the team tested the method
The work process was divided into the following stages:
in the exhibition facility themselves, it soon became evident that plans

had to be simplified in order to be able to record comments.
- defining needs and objectives

- planning the implementation
After the first meeting, preparations began for a recording form that
- preparing for implementation
would guide recorders to make mutually comparable observations. After
- implementation
the elimination of other features, only two targets of observation
- processing and analysing the material
remained:
- reporting

 visitor’s route and
The entire process has been an inspiring learning and development
experience for the writers of the report.  feelings expressed by the visitor (positive and negative
comments)
4.1. Planning stage

The idea to organise an evaluation workshop was born in June 2010 when
the Finnish Museums Association proposed cooperation with the Espoo
City Museum to organise the Exhibition Café. The writers of the report
have engaged in informal professional discussions on the development of
exhibition planning already since the Exhibition Café of 2007. The
- 5 -
The form could thus be shortened to two pages. The aim was to make it
as simple and clear as possible (Images 1a and 1b). The form contained:

 The exhibition layout where the recorder could draw the group’s
route
 Plus and minus columns for comments
 Brief instructions for recorders

All willing employees of the City Museum were asked to take part as
recorders. They participated in one instructive meeting before the day of
the workshop.

Image 1b: Recorders’ form, page 2

Image 1a: Recorders’ form, page 1

- 6 -
4.2. In the workshop

The large group of participants at the training event (51 people) had to be
divided into smaller groups for the workshop. In order to bring out
different perspectives, participants were asked to fill in a questionnaire
(Image 2), based on which they were divided into different types of visitor
groups.

Participants filled in the questionnaire in the morning of the seminar day,
after which they were divided into groups of 4–5 people. There were a
total of 10 groups that were each given their own role:

1 group first-timers at the WeeGee Exhibition Centre, in the Espoo
City Museum
3 groups visiting families and those interested in accessibility
2 groups those working with groups of pupils (role: teacher who is
e.g. thinking about bringing her class to the exhibition)
4 groups those interested in exhibition technology and structures

After the division, each group was appointed their personal recorder who
gave instructions and described the role assigned to the group. The
groups were also reminded that making comments and speaking out loud
was the most important thing, not strict adherence to the role.

The recorders’ task was to write down the visitors’ comments into the
plus and minus columns of the form and draw the group’s route onto the
exhibition layout. Additionally, their task was to mark + and – signs along
the route, based on the group’s comments. Eight of the recorders were
part of the City Museum’s own staff and two were from the Finnish
Museums Association. It was emphasised that recorders were not
supposed to guide the small group or answer questions concerning the
content or organisation of the exhibition. However, the aim was to
achieve as natural a presence as possible. Test visitors were asked to turn
to museum guides if they had any questions, as they would during an
Image 2: Questionnaire based on which the participants were divided into
ordinary museum visit.
separate groups with roles in the workshop.
- 7 -
After receiving instructions, the groups entered the exhibition from the During the short break provided to participants, the coordinators
lobby of the WeeGee Exhibition Centre at a staggered pace. The group compiled a quick summary of the results of the evaluation workshop: the
that was visiting WeeGee or the City Museum for the first time was given plusses and minuses of the groups (Images 3 and 4) were compiled
a starting point outside the main entrance to the Exhibition Centre. The together onto individual layouts and the development proposals were
aim of this was to gather information also on how a new visitor listed on a single form (Image 5).
distinguishes KAMU from the five museums in the WeeGee Exhibition
Centre. Groups were given 45 minutes to tour the exhibition. Two
workshop coordinators circled around observing the movement of groups
and recorders in the exhibition facility. The atmosphere was lively and
crowded. Everyone took on the task seriously and actively.

4.3. Immediate observations and feedback discussion in the
Exhibition Café
After the tour, group recorders returned their forms to the coordinators.
Each group was simultaneously interviewed briefly and asked to mention
one development target and one best feature in the exhibition.

Image 4: Minus signs (= negative comments by groups) compiled from the
recorders’ forms onto the exhibition layout.

Image 3: Plus signs (= positive comments by groups) compiled from the

recorders’ forms onto the exhibition layout.

- 8 -
The immediate results of the evaluation workshop were presented to the
participants in the feedback discussion held after the break. The material
generated by the evaluation indicated that:

 The plusses and minuses gathered from the groups’ comments
onto the layout appeared to be focused particularly on two
places: minuses at the start of the exhibition and pluses at the
material showcase / touch screens / material samples near the
middle of the exhibition.
 Comments (1 best feature / 1 development target) elicited from
groups were similar. They clearly showed one minus and one plus
feature that are in direct correlation with the markings on the
layouts:
+ showcase of clothes + material samples + material
information on the touch screen is a functional hands-on
ensemble
- introduction is missing, the beginning is difficult to grasp

During the feedback discussion, the following aspects, among others,
were brought up by workshop participants:

 the division into separate role groups did not particularly bring
additional value to the evaluation
 the evaluators, or those thinking aloud, found the method
interesting and the experience rewarding
 recorders likewise found the experience positive, but also
discovered ideas for improvement
 the method, implemented as it was, provided clear information
on the basic experiences of visitors and a strong evidence on the
most important development targets of the exhibition

Image 5: Best features and development targets of the exhibition mentioned by
Finally, the audience was briefly introduced to the background of the
the groups
Thinking Aloud method and ideas on its practical application.

- 9 -
4.4. Subsequent processing and analysis of evaluation material

The material collected at the workshop was examined in more detail
during October–December. A written analysis for each exhibition section
(pages 11–12) was drawn up, based on the forms filled in by recorders.
Recorders were also interviewed afterwards in order to clarify the
handwriting and notes that were jotted down quickly. The routes drawn
by recorders were redrawn clearly, and the plus and minus signs marked
on the maps were compiled onto a single layout (Images 6, 7 and 8). Even
prior to written analysis, the compilation and the route drawings indicate
that the method succeeded in gathering useful and practical information
on the strengths and weaknesses of the exhibition based on visitors’
experiences. The written analysis provided more detailed information on
the reasons behind the positive and negative experiences of visitors.

Image 6: A compilation of the positive and negative feelings expressed by visitors.

Image 7: One of the evaluation groups followed the route that the exhibition
planners had intended.

- 10 -