Experiences with usability testing

205 Pages·2015·4.7 MB·English

by  Sirpa Riihiaho

Department of Computer Science
Experiences with usability
Usability testing has become a standard A S
method when evaluating the usability of alto-D irpa
various systems with real users. Despite its D 75 Rii testing
goef nuesraabli ulistye taensdt iwngid hea avcec beepetnan gcivee, tnh lei tftalcet ors /2015 hiah
o

attention in academic forums. This thesis
studies methods and factors of usability E
x Effects of thinking aloud and moderator presence
p
testing focusing on the effects of thinking e
r
aloud and the presence of a test moderator. ie
n
It combines an extensive literature review c
e
s
with experiences of usability testing from 22 w
i
years covering 143 usability studies. It also th u Sirpa Riihiaho
includes an experimental part with relaxed s
a
thinking aloud and the presence of a test b
i
l
i
moderator as independent variables. t
y
t
e
s
t
i
n
g

ISBN 978-952-60-6226-6 (printed) BUSINESS +
9 ISBN 978-952-60-6227-3 (pdf) ECONOMY
H ISSN-L 1799-4934
S ISSN 1799-4934 (printed) ART +
T
F ISSN 1799-4942 (pdf) DESIGN +
M ARCHITECTURE
G* Aalto University Aa
agcc SDwcewhpwoa.oartlam olteof n.Sftic oief nCcoem puter Science STECCIEHNNCOEL +O GY lto Un
gg CROSSOVER iv
+ er DOCTORAL
DOCTORAL s DISSERTATIONS
DISSERTATIONS ity
Aalto University publication series
DOCTORAL DISSERTATIONS 75/2015
Experiences with usability testing:
Effects of thinking aloud and moderator presence
Sirpa Riihiaho
A doctoral dissertation completed for the degree of Doctor of
Science (Technology) to be defended, with the permission of the
Aalto University School of Science, at a public examination held at
the lecture hall T2 of the school on 26th June 2015 at 12 noon.
Aalto University
School of Science
Department of Computer Science
Strategic Usability Research Group
Supervising professor
Marko Nieminen

Preliminary examiners
Sharon McDonald, University of Sunderland, United Kingdom
Jan Stage, Aalborg University, Denmark

Opponent
Judith A. Ramey, University of Washington, United States
Aalto University publication series
DOCTORAL DISSERTATIONS 75/2015

© Sirpa Riihiaho

ISBN 978-952-60-6226-6 (printed)
ISBN 978-952-60-6227-3 (pdf)
ISSN-L 1799-4934
ISSN 1799-4934 (printed)
ISSN 1799-4942 (pdf)
http://urn.fi/URN:ISBN:978-952-60-6227-3

Unigrafia Oy
Helsinki 2015

Finland

Publication orders (printed book):
[email protected]
Abstract
Aalto University, P.O. Box 11000, FI-00076 Aalto www.aalto.fi
Author
Sirpa Riihiaho
Name of the doctoral dissertation
Experiences with usability testing: Effects of thinking aloud and moderator presence
Publisher School of Science
Unit Department of Computer Science
Series Aalto University publication series DOCTORAL DISSERTATIONS 75/2015
Field of research Computer Science and Engineering
Manuscript submitted 15th April 2014 Date of the defence 26th June 2015
Permission to publish granted (date) 8th April 2015 Language English
Monograph Article dissertation (summary + original articles)
Abstract
Usability testing has become a standard method when evaluating the usability of various
systems with real users. Despite this, the factors of usability testing have been given little atten-
tion in the academic forums. For example, the use of the thinking aloud method has resulted in
quite conflicting effects on the users' performance in the context of usability testing, and the
effects of the moderator presence have been studied very seldom in usability research.
This thesis studies methods of usability testing and contextual factors in the test settings that
may affect the results of a test, focusing on the effects of relaxed thinking aloud and the pres-
ence of a test moderator. It combines an extensive literature review with experiences on usa-
bility testing from 22 years covering 143 usability studies. The challenges of usability testing
reported in the top academic HCI forums focus on sampling, context of use, use over time, and
assessment of utility and value. The methods presented in this thesis provide some solutions
to problems related to the context of use and to the assessment of utility and value.
The most significant contribution of this thesis is in the experiment with thinking aloud and
moderator presence as its independent variables. The results of relaxed thinking aloud show
that it has no significant effect compared to silent performance on the number of usability
problems the users face or in their subjective ratings, but it does slow down their performance.
A significant effect of the moderator presence is found in the users' subjective rating, as users
with a moderator next to them rate the system preferences significantly higher than partici-
pants performing alone. Given the benefits of having a moderator next to the user who is able
to ask clarifying questions when the experiences are fresh in user's mind, the thesis still rec-
ommends this approach in formative usability testing when it is important to come up with
practical redesign proposals for the development team.
Thinking aloud in this experiment does not enhance the evaluators' confidence in the de-
tected usability problems and their causes, and the test users mostly report it as an unnatural
extra effort. Even so, thinking aloud gives more information about the problems to customers
observing the tests, and thereby may motivate designers to make the required changes to the
system. Therefore, if performance measurements are required, silent performance or classic
concurrent thinking aloud with minimal interventions should be used, but in formative testing,
the more explanatory relaxed thinking aloud can be used, as long as its potential effects on
users' performance are kept in mind.

Keywords usability testing, thinking aloud, moderator presence, usability evaluation methods
ISBN (printed) 978-952-60-6226-6 ISBN (pdf) 978-952-60-6227-3
ISSN-L 1799-4934 ISSN (printed) 1799-4934 ISSN (pdf) 1799-4942
Location of publisher Helsinki Location of printing Espoo Year 2015
Pages 200 urn http://urn.fi/URN:ISBN:978-952-60-6227-3

Tiivistelmä
Aalto-yliopisto, PL 11000, 00076 Aalto www.aalto.fi
Tekijä
Sirpa Riihiaho
Väitöskirjan nimi
Kokemuksia käytettävyystestauksesta: Ääneenajattelun ja ohjaajan läsnäolon vaikutuksia
Julkaisija Perustieteiden korkeakoulu
Yksikkö Tietotekniikan laitos
Sarja Aalto University publication series DOCTORAL DISSERTATIONS 75/2015
Tutkimusala Tietotekniikka
Käsikirjoituksen pvm 15th April 2014 Väitöspäivä 26th June 2015
Julkaisuluvan myöntämispäivä 8th April 2015 Kieli Englanti
Monografia Yhdistelmäväitöskirja (yhteenveto-osa + erillisartikkelit)
Tiivistelmä
Käytettävyystestauksesta on tullut vakiintunut käytäntö erilaisten järjestelmien käytettä-
vyyden arviointiin yhdessä käyttäjien kanssa. Tästä huolimatta käytettävyystestauksen osa-
tekijät ovat jääneet hyvin pienelle huomiolle akateemisessa keskustelussa. Ääneenajattelun
vaikutuksista on tutkimuksissa saatu hyvin ristiriitaisia tuloksia käytettävyystestauksessa, ja
ohjaajan läsnäolon vaikutuksia on tutkittu hyvin vähän käytettävyysalalla.
Tässä opinnäytteessä tutkitaan käytettävyystestauksen menetelmiä ja osatekijöitä, jotka
voivat vaikuttaa testauksen tuloksiin, keskittyen rentoutuneen ääneenajattelun ja ohjaajan
läsnäolon vaikutuksiin. Työssä yhdistyy laaja kirjallisuuskatsaus ja 22 vuoden kokemukset 143
käytettävyysarviosta. Alan huippujulkaisuissa raportoidut käytettävyystestauksen haasteet
keskittyvät testiotoksiin, käyttötilanteeseen, pidempiaikaiseen käyttöön ja hyödyn ja arvon
arviointiin. Opinnäytteessä esitellyt menetelmät tarjoavat ratkaisuja näihin ongelmiin käyttö-
tilanteen ja hyödyn ja arvon arvioinnin osalta.
Tämän opinnäytteen tärkein anti on kokeellisessa osiossa, jossa riippumattomina muuttujina
ovat ääneenajattelu ja testiohjaajan läsnäolo. Rentoutuneella ääneenajattelulla ei hiljaiseen
toimintaan verrattuna ole merkittävää vaikutusta käyttäjien kohtaamien käytettävyysongel-
mien määrään tai heidän miellyttävyysarvioihinsa, mutta se hidastaa heidän toimintaansa.
Testiohjaajan läsnäololla sen sijaan on merkittävä vaikutus, sillä ohjaajan läsnäollessa toimi-
neet käyttäjät arvioivat järjestelmän miellyttävämmäksi kuin yksin olleet. Vaikka ohjaajan
läsnäoloon liittyy riski saada positiivisempia arvioita, opinnäyte suosittelee ohjaajan läsnäoloa
etenkin tuotekehityksen aikaisessa testauksessa, jossa etsitään keinoja parantaa tuotteen
käytettävyyttä. Kun ohjaaja on käyttäjän lähellä, hänellä on erinomainen mahdollisuus kysyä
tarkentavia kysymyksiä kunkin tehtävän jälkeen kokemusten ollessa vielä käyttäjän tuoreessa
muistissa.
Ääneenajattelu ei lisää tässä koeasetelmassa arvioijien varmuutta löytämistään ongelmista
tai niiden syistä, ja käyttäjät kokevat ääneenajattelun lähinnä luonnottomana lisäponnisteena.
Ääneenajattelu kuitenkin antaa testejä seuraavalle asiakkaalle lisätietoa ongelmista ja voi täten
motivoida tekemään vaaditut muutokset. Niinpä on suositeltavaa käyttää perinteistä ääneen-
ajattelua mahdollisimman pienin keskeytyksin suoritusmittauksia sisältävässä testauksessa,
mutta kehityksen aikaisessa testauksessa voidaan käyttää selittävämpää rentoutunutta
ääneenajattelua, kunhan sen mahdolliset vaikutukset käyttäjien toimintaan tiedostetaan.
Avainsanat käytettävyystestaus, ääneenajattelu, ohjaajan läsnäolo, käytettävyyden
arviointimenetelmät
ISBN (painettu) 978-952-60-6226-6 ISBN (pdf) 978-952-60-6227-3
ISSN-L 1799-4934 ISSN (painettu) 1799-4934 ISSN (pdf) 1799-4942
Julkaisupaikka Helsinki Painopaikka Helsinki Vuosi 2015
Sivumäärä 200 urn http://urn.fi/URN:ISBN:978-952-60-6227-3

Preface
Gathering the evaluation data and writing this thesis has taken a long time, so
there have been several projects and funders that have affected my research,
and many persons that have been involved in the process during these years.
In the early phases of my research, Leo and Regina Wainstein's Foundation
and Tekniikan edistämissäätiö supported my work financially with grants.
Also the Academy of Finland, the National Technology Agency (TEKES) and
the Finnish Work Environment Fund (Työsuojelurahasto) have funded my
research. The experimental part of this thesis, on its part, was conducted in the
Mobile Financial Services (MoFS) project funded by TiViT/Digile.
Many colleagues and students have participated in the evaluations that
have affected this thesis, and several persons have helped in finding the time
for my research and writing. Friends and teammates in Niitti have also been
very important for me in finding some balance between my work and hobbies.
I am grateful to all of you. I want to especially thank my supervising professor
Marko Nieminen for his support, all the active students trying out my ideas in
their course assignments, and all my current and former colleagues for inter-
esting discussions on life, the Universe and everything – including usability.
Finally, my warmest thanks go to my family. My mother and my already de-
ceased father have always supported me in my studies and life, and also my
sister has always helped me in numerous ways. But most of all, my devoted
husband Jukka and lovely daughter Sanni have been the central part of my
life: thank you for sharing the joy and misery of life with me!

Sirpa Riihiaho
Vantaa, May 1st 2015

i(cid:1)
Contents
1(cid:1) Introduction.........................................................................................1(cid:1)
1.1(cid:1) Nature and scope of this thesis.....................................................3(cid:1)
1.2(cid:1) Research problem and questions.................................................5(cid:1)
1.3(cid:1) Field of studies..............................................................................5(cid:1)
2(cid:1) Research methods and material.........................................................7(cid:1)
2.1(cid:1) Literature review...........................................................................7(cid:1)
2.1.1(cid:1) Usability and usability testing in ACM publications.............8(cid:1)
2.1.2(cid:1) Top HCI forums......................................................................9(cid:1)
2.1.3(cid:1) Usability in top HCI journals...............................................11(cid:1)
2.1.4(cid:1) Usability testing in HCI conferences...................................14(cid:1)
2.1.5(cid:1) Activity of HCI forums in usability issues............................15(cid:1)
2.2(cid:1) Empirical usability studies.........................................................17(cid:1)
2.2.1(cid:1) Usability evaluation process in Aalto University.................19(cid:1)
2.2.2(cid:1) Empirical data......................................................................21(cid:1)
2.3(cid:1) Experimental research..............................................................24(cid:1)
2.3.1(cid:1) Designing experiments........................................................24(cid:1)
2.3.2(cid:1) Analysis of experiments.......................................................25(cid:1)
2.3.3(cid:1) Reliability and validity........................................................26(cid:1)
3(cid:1) Usability testing...............................................................................28(cid:1)
3.1(cid:1) Process of usability testing........................................................30(cid:1)
3.1.1(cid:1) Planning usability evaluations..............................................31(cid:1)
3.1.2(cid:1) Test moderator.....................................................................31(cid:1)
3.1.3(cid:1) Test participants..................................................................32(cid:1)
3.1.4(cid:1) Number of test users............................................................33(cid:1)
3.1.5(cid:1) Test environment and use context......................................34(cid:1)
3.1.6(cid:1) Usability metrics.................................................................36(cid:1)
3.1.7(cid:1) Test tasks and scenario.........................................................37(cid:1)
3.1.8(cid:1) Questionnaires and interviews............................................37(cid:1)
3.1.9(cid:1) Conducting tests..................................................................39(cid:1)
ii(cid:1)
Contents
3.1.10(cid:1) Analysis of test sessions......................................................40(cid:1)
3.1.11(cid:1) Communicating results........................................................41(cid:1)
3.2(cid:1) Thinking aloud method.............................................................43(cid:1)
3.2.1(cid:1) Effects on performance in problem solving.........................45(cid:1)
3.2.2(cid:1) Thinking aloud in usability testing......................................47(cid:1)
3.2.3(cid:1) Concurrent thinking aloud.................................................49(cid:1)
3.2.4(cid:1) Retrospective thinking aloud...............................................52(cid:1)
3.2.5(cid:1) Effect of thinking aloud instructions...................................54(cid:1)
3.3(cid:1) Modifications of usability testing...............................................55(cid:1)
3.3.1(cid:1) Question asking protocol.....................................................56(cid:1)
3.3.2(cid:1) Cooperative evaluation........................................................56(cid:1)
3.3.3(cid:1) Cooperative Usability Testing..............................................57(cid:1)
3.3.4(cid:1) Critical incidents and backtracking analysis......................58(cid:1)
3.3.5(cid:1) Experience Clip....................................................................59(cid:1)
3.4(cid:1) Usability inspection methods....................................................60(cid:1)
3.5(cid:1) Criteria for assessing usability evaluation methods..................61(cid:1)
3.6(cid:1) Impact on development process...............................................62(cid:1)
3.7(cid:1) Experiments on contextual factors of usability testing............64(cid:1)
3.7.1(cid:1) Participating an experiment.................................................65(cid:1)
3.7.2(cid:1) Test users’ expertise.............................................................65(cid:1)
3.7.3(cid:1) Test users’ expectations......................................................66(cid:1)
3.7.4(cid:1) Test environment.................................................................67(cid:1)
3.7.5(cid:1) Moderator presence............................................................70(cid:1)
3.7.6(cid:1) Prototype level......................................................................71(cid:1)
3.7.7(cid:1) System aesthetics.................................................................73(cid:1)
3.7.8(cid:1) Evaluator effect....................................................................75(cid:1)
3.8(cid:1) Challenges in usability testing...................................................76(cid:1)
3.8.1(cid:1) Sampling users and test tasks..............................................76(cid:1)
3.8.2(cid:1) Context of use, use over time and utility assessment.........78(cid:1)
3.8.3(cid:1) Misuses of usability testing................................................78(cid:1)
3.8.4(cid:1) Dogmas in assessment of methods......................................79(cid:1)
3.9(cid:1) Effects of literature review on this thesis..................................80(cid:1)
4(cid:1) Modifications of usability testing in Aalto University......................81(cid:1)
4.1(cid:1) Paired-user testing.....................................................................82(cid:1)
4.1.1(cid:1) Experiences with paired-user testing..................................83(cid:1)
4.1.2(cid:1) Experiences with peer tutoring...........................................84(cid:1)
4.2(cid:1) Pluralistic usability walkthrough..............................................85(cid:1)
iii