UNIVERSITY OF CALGARY Distributed Wizard of Oz Usability Testing for Agile Teams by ALI ...

225 Pages·2009·8.67 MB·English

by  Alessandro Del Piero

UNIVERSITY OF CALGARY

Distributed Wizard of Oz Usability Testing for Agile Teams

by

ALI HOSSEINI KHAYAT

A THESIS
SUBMITTED TO THE FACULTY OF GRADUATE STUDIES
IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE
DEGREE OF MASTER OF COMPUTER SCIENCE

DEPARTMENT OF COMPUTER SCIENCE
CALGARY, ALBERTA
DECEMBER 2009

© Ali Hosseini Khayat 2009
UNIVERSITY OF CALGARY
FACULTY OF GRADUATE STUDIES

The undersigned certify that they have read, and recommend to the Faculty of Graduate
Studies for acceptance, a thesis entitled "Distributed Wizard of Oz Usability Testing for
Agile Teams" submitted by ALI HOSSEINI KHAYAT in partial fulfillment of the
requirements of the degree of Master of Science.

Supervisor, Dr. Frank Oliver Maurer, Department of Computer
Science

Dr. Richard Levy, Faculty of Environmental Design

Dr. Elaine Huang, Faculty of Computer Science

Date

ii
Abstract
Distributed usability testing has the potential to be a valuable tool for agile usability
practitioners, however it is unclear what type of usability data can be collected and which
representations of it are useful for detecting flaws. Conducting Wizard of Oz testing in a
distributed fashion over the web can significantly reduce the cost implications of usability
testing as well as the time commitment necessary. This is of particular interest to agile
methods where iterations are short and frequent and minimal upfront design is prescribed.
Traditional usability testing can be prohibitively expensive and time consuming as it
requires large amounts of upfront design. Moreover, the process of recruiting participants
and bringing them on site is costly both in terms of time and financial resources.
Distributed Wizard of Oz testing is an alternative that allows cheap and quick usability
testing of low-fidelity prototypes over the web.
The most important output from a usability test is the feedback regarding the system
under test. In traditional usability testing, a human observer directly collects this data by
studying the test user’s actions. However, when conducting a distributed usability test,
specifically a Wizard of Oz test on a low-fidelity prototype, it is not possible for a human
observer to collect this data. Instead the Wizard of Oz testing system must automatically
observe the user and capture relevant usability data. This thesis evaluates multiple forms
of automatically collected usability data as well as visual representations of it in terms of
their efficiency and usefulness in aiding in the detection of usability flaws. It also
presents ActiveStory Enhanced, a tool designed to support the entire agile usability
testing process from prototyping to testing to analysis. ActiveStory Enhanced is available
for download on CodePlex (Link [1])
iii
Acknowledgements
This research and thesis are the culmination of help, guidance and support from many
people. I would like to acknowledge and thank them for their roles in getting me here.
To Dr. Maurer, thanks for the invaluable feedback and advice, without which this thesis
would not be possible.
To all of my friends in the Agile Software Engineering lab and elsewhere, Kunnal
Khiatani, Fadl Obeid, Theodore Hellmann, Navid Delshad and others. Thanks for being
great friends and encouraging me along the way.
To Bill Gates and Microsoft, who are the reasons I got interested in computers and
programming in the first place. Thanks for making it possible to have a PC in every
home.
Finally, thanks to my Father for supporting my ambition to pursue a career in the field of
computers. Thanks to my Mother for buying that first PC, I wouldn’t have been here
without it. Thanks to my family, Mother, Father, Mohsen, Atieh and Noah, for supporting
and encouraging me throughout my entire education.
iv
Dedication
To God and my family for always supporting me in every way.
v
Publications from this Thesis
Portions of the materials and ideas presented in this thesis may have appeared previously
in the following peer reviewed publications:
 Ali Hosseini-Khayat, Yaser Ghanam, Shelly Park and Frank Maurer,
“ActiveStory Enhanced: Low Fidelity Prototyping and Wizard of Oz Usability
Testing Tool” in Proceedings of the 10th International Conference on Agile
Processes and eXtreme Programming (XP 2009), Demo Abstract, Pula, Italy,
2009.
 Patrick Wilson, Ali Hosseini-Khayat, David Fox and Frank Maurer, “ActiveStory:
A Tool for Agile – UCD Design Teams”, Agile 2009 Workshop on Research-in-
Progress, Chicago, Illinois, 2009.
vi
Table of Contents
Chapter One: Introduction ................................................................................................ 18 
1.1 Usability and its Importance ................................................................................... 18 
1.2 Agile Methods, Conflicts with Usability Engineering ............................................ 19 
1.3 Agile Usability ........................................................................................................ 20 
1.4 Tool Support for Agile Usability ............................................................................ 23 
1.5 Usability Data and its Visualizations ...................................................................... 24 
1.6 Motivation Summary............................................................................................... 25 
1.7 Research Problem .................................................................................................... 25 
1.8 Goals........................................................................................................................ 26 
1.9 Thesis Structure ....................................................................................................... 26 
Chapter Two: Related Work ............................................................................................. 28 
2.1 Practical Agile Interaction Design .......................................................................... 28 
2.1.1 Larry Constantine ............................................................................................. 29 
2.1.2 Jeff Patton ......................................................................................................... 30 
2.1.3 Desiree Sy ......................................................................................................... 31 
2.1.4 Gerard Meszaros ............................................................................................... 33 
2.1.5 Jennifer Ferreira ................................................................................................ 34 
2.1.6 Helen Sharp ...................................................................................................... 35 
2.1.7 Paul McInerney and Frank Maurer ................................................................... 36 
vii
2.1.8 David Fox ......................................................................................................... 36 
2.1.9 Jakob Nielsen .................................................................................................... 37 
2.2 Remote Usability Evaluation .................................................................................. 38 
2.3 Existing Tool Support for Prototyping .................................................................... 40 
2.3.1 Pen and Paper ................................................................................................... 42 
2.3.2 Microsoft Expression Blend – SketchFlow ...................................................... 43 
2.3.3 ActiveStory ....................................................................................................... 46 
2.3.4 SILK ................................................................................................................. 49 
2.3.5 DENIM ............................................................................................................. 49 
2.3.6 Serena Prototype Composer ............................................................................. 50 
2.3.7 Balsamiq Mockups ........................................................................................... 51 
2.3.8 Microsoft PowerPoint ....................................................................................... 52 
2.3.9 Microsoft Visio ................................................................................................. 53 
2.3.10 Intuitect ........................................................................................................... 53 
2.3.11 AxureRP ......................................................................................................... 53 
2.3.12 WoZPro .......................................................................................................... 55 
2.4 Existing Tool Support for Usability Testing ........................................................... 57 
2.4.1 Wizard of Oz Testing ....................................................................................... 57 
2.4.2 OzLab ............................................................................................................... 58 
2.4.3 Neimo ............................................................................................................... 59 
viii
2.4.4 Morae ................................................................................................................ 60 
2.5 Summary ................................................................................................................. 61 
Chapter Three: Tool Requirements ................................................................................... 62 
3.1 Gathering Requirements .......................................................................................... 62 
3.2 Requirements Gathered from Survey ...................................................................... 63 
3.2.1 Easy to Use ....................................................................................................... 63 
3.2.2 Pen and Paper Metaphor ................................................................................... 64 
3.2.3 Fast Creation and Testing ................................................................................. 64 
3.2.4 Design Flexibility ............................................................................................. 64 
3.2.5 Interactive Prototypes ....................................................................................... 64 
3.2.6 Remote, Distributed Testing ............................................................................. 65 
3.2.7 Data Collection from Distributed Tests ............................................................ 65 
3.3 Requirements Extracted from Evaluation of Original ActiveStory ........................ 66 
3.3.1 Evaluation Summary ........................................................................................ 66 
3.3.2 Extracted Requirements .................................................................................... 68 
3.4 Summary ................................................................................................................. 69 
Chapter Four: Usability Data, Visualizations and Analysis ............................................. 70 
4.1 Usability Data and Visualization Candidates .......................................................... 70 
4.1.1 Usability Data ................................................................................................... 71 
4.1.2 Visualizing Usability Data................................................................................ 78 
ix
4.1.3 Analysis ............................................................................................................ 86 
4.2 Usability Data and Visualizations in an Agile Context ........................................... 88 
4.2.1 Usability Data ................................................................................................... 88 
4.2.2 Visualizations and Data Analysis ..................................................................... 92 
4.3 Summary ................................................................................................................. 97 
Chapter Five: ActiveStory Enhanced ................................................................................ 98 
5.1 Toolset Overview .................................................................................................... 98 
5.2 Prototype Designer Overview ............................................................................... 100 
5.2.1 Project and Pages ............................................................................................ 101 
5.2.2 Drawing .......................................................................................................... 102 
5.2.3 Interactions ..................................................................................................... 105 
5.2.4 Exporting ........................................................................................................ 107 
5.3 Wizard of Oz Testing Tool Overview ................................................................... 108 
5.3.1 Participants ..................................................................................................... 109 
5.3.2 Prototype Evaluation ...................................................................................... 110 
5.4 Test Reports Tool Overview ................................................................................. 112 
5.4.1 Mouse Trails and Clicks ................................................................................. 113 
5.4.2 Live Session Playback .................................................................................... 114 
5.4.3 Aggregate Reports .......................................................................................... 115 
5.4.4 Statistic Charts ................................................................................................ 116 
x