Improving Requirements Engineering and Usability Testing Practices for Agile SaaS Development

99 Pages·2017·4.76 MB·English

Hanna-Mari Kinnunen

Improving Requirements Engineering and Usability Testing
Practices for Agile SaaS Development

Thesis submitted for examination for the degree of Master of
Science in Technology.

Espoo 22.05.2017
Supervisor: Professori Marjo Kauppinen
Instructor: Niila Gauriloff
Aalto University, P.O. BOX 11000, 00076
AALTO

www.aalto.fi
Abstract of master's thesis

Author Hanna-Mari Kinnunen
Title of thesis Improving Requirements Engineering and Usability Testing Practices for
Agile SaaS development
Degree programme Communication and Information Sciences
Major/minor Software and Service Engineering Code SCI3043
Thesis supervisor Marjo Kauppinen
Thesis advisor(s) Niila Gauriloff
Date 22.05.2017 Number of pages 95 Language English
Both requirements engineering and usability testing practices are well documented in the
literature, but implementing them in real life as a part of agile software development is not self-
evident. The situation gets even more complicated when the customer base of the product is
diverse. This thesis seeks solutions for research problem of how requirements can be defined and
managed in a SaaS (Software as a Service) product that has multiple customer segments.

The goal for the literature review was to find good practices for requirements engineering and
usability testing that could be implemented to the development of the target product. Empirical
study was done by using insider action research method. Several of the suggested good practices
and activities were tried out and the results analysed.

The empirical study begun with customer segmentation during the current state analysis. It was
used as a basis during the rest of the project. Four separate customer collaboration methods were
tested: survey, interviewing, observing and user testing. The survey was a very good method for
getting large amount of both qualitative and quantitative data about the usability and
requirements of the target product. Interviewing and user testing gave more insights to
workflows and business models of the customers. Furthermore, they revealed some missing
features and usability issues about the product. Observing gave some insight about the
customer’s use of the software but it did not seem to be as informative than expected. Important
success factor for this project was to assign a dedicated person inside the company who is
responsible for requirements management and usability - the product owner. That change
enabled implementing the use of scrum in the development of the target software.

Conclusion is that the complex customer base of the product can be analysed and managed by
creating a customer segmentation and to use it as a basis for all the requirements engineering and
usability testing practices. The product owner can facilitate these activities and it is advisable to
use multiple customer collaboration methods to gain as thorough big picture as possible.

Keywords Requirements Engineering, SaaS, Usability Testing, Usability, Agile,
customer segmentation
Aalto-yliopisto, PL 11000, 00076
AALTO

www.aalto.fi
Diplomityön tiivistelmä

Tekijä Hanna-Mari Kinnunen
Työn nimi Improving Requirements Engineering and Usability Testing practices for Agile
SaaS development
Koulutusohjelma Master's Programme in Computer, Communication and Information
Sciences
Pää-/sivuaine Software and Service Engineering Koodi SCI3043
Työn valvoja Marjo Kauppinen
Työn ohjaaja(t) Niila Gauriloff
Päivämäärä 22.05.2017 Sivumäärä 95 Kieli englanti
Sekä vaatimusmäärittely että käytettävyystestaus on hyvin määritelty ja dokumentoitu
kirjallisuudessa, mutta niiden soveltaminen ketterään tuotekehitykseen ei ole itsestään selvää.
Tilanne muuttuu entistä haastavammaksi, kun tuotteen asiakaskunta sisältää monipuolisen kirjon
eri tyyppisiä organisaatioita. Tämä diplomityö etsii ratkaisuja tutkimusongelmaan “kuinka
vaatimuksia voidaan määritellä ja hallita SaaS-tuotteessa (Software as a Service), jolla on monia
eri asiakassegmentteja”.

Kirjallisuuskatsauksen tavoite oli löytää hyviä käytäntöjä kohdetuotteen vaatimusmäärittelyyn ja
käytettävyystestaukseen. Empiirinen osa toteutettiin toimintatutkimuksena, jonka toteuttaja oli
tämän työn tekijä. Useita ehdotettuja hyviä toimintatapoja ja menetelmiä kokeiltiin käytäntöön
työn aikana ja saadut tulokset analysoitiin.

Empiirinen tutkimus aloitettiin luomalla asiakassegmentointi osana lähtötilan analyysia. Koko
loppuprojekti suunniteltiin tämän tiedon pohjalta. Neljää eri menetelmää asiakasyhteistyöhön
kokeiltiin: kysely, haastattelu, havainnointi ja käytettävyystestaus. Kyselystä saatiin suuri määrä
sekä kvalitatiivista että kvantitatiivista tietoa käytettävyydestä ja tuotteen vaatimuksista.
Haastattelu ja käytettävyystestaus tuottivat lisätietoa kohdeasiakkaiden työprosesseista ja
organisaatioiden toiminnasta. Näiden menetelmien käyttö toi esiin myös puuttuvia ominaisuuksia
ja käytettävyyshavaintoja. Havainnointi antoi lisätietoa asiakkaan ohjelmiston käytöstä, mutta se
ei vaikuttanut niin hyödylliseltä menetelmältä kuin kirjallisuuden perusteella odotettiin.
Kohdeorganisaatiossa otettiin tuoteomistaja-rooli käyttöön täysipainoisesti, jonka vastuulla on
vaatimusmäärittely- ja käytettävyystestausprosessit. Tämä muutos myös mahdollisti scrumin
täysipainoisen käyttöönoton kehitysprosessiksi.

Työn johtopäätös on, että asiakassegmentointi voi auttaa tuotteen monipuolisen asiakaskunnan
jäsentelyssä ja sitä pystyy myös käyttämään hyväksi vaatimusmäärittelyn ja
käytettävyystestauksen menetelmiä käytettäessä. Tuoteomistaja on vastuussa näiden menetelmien
käytöstä ja hänen kannattaa käyttää useita eri asiakasyhteistyön menetelmiä, jotta hän saavuttaa
mahdollisimman hyvän kokonaiskuvan tuotteesta ja siihen liittyvistä vaatimuksista.

Avainsanat Vaatimusmäärittely, SaaS, käytettävyystestaus, käytettävyys, ketteryys,
asiakassegmentointi
Acknowledgement

Finally, this thesis is finished. Everything started over a year ago when we started to plan
the topic for my thesis with my colleague Jarkko Lakso. At first the whole topic was
really abstract and hard to grasp. In time the goal of this project grew into a real change
process. There were times when all I tried to accomplish seemed next to impossible. Still,
I didn’t give up. It was worth the effort, finally I got the privileges to make all the
necessary changes to current practices and it is amazing to see the effects in practice. Now
this thesis is done but my work continues.

At first I want to thank my supervisor Marjo Kauppinen for believing in me and
supporting me during the whole process. This thesis wouldn’t exist without Jarkko Lakso,
thank you for trusting me with this topic. Also, my instructor and colleague Niila
Gauriloff had a significant role throughout this project. He was always ready to hear my
worries, to read and comment my text and to suggest improvements. I also want to thank
IMS Business Solutions Oy (Arter Oy) for giving me privilege to do this project and all
my colleagues for supporting me on the way. At times, it has been challenging to find
time for this work. I would never had graduated without my family – my dear husband
Olli and our parents. Thank you for giving me the time to study and write. Graduating
and finishing this thesis means a world to me.

Espoo 22.5.2017

Hanna-Mari Kinnunen
Table of Contents

1 Introduction ............................................................................................................. 3
1.1 Motivation ........................................................................................................ 3
1.2 Research problem and questions ................................................................... 4
1.3 Scope and focus of the thesis .......................................................................... 4
1.4 Structure of the thesis ..................................................................................... 5
2 Agile and Lean methodology ................................................................................. 7
2.1 Agile and Lean in short .................................................................................. 7
2.2 Scrum ............................................................................................................... 7
2.3 Lean in Software Development ...................................................................... 8
2.4 Agile SaaS development ................................................................................. 9
3 Requirements Engineering ................................................................................... 11
3.1 Definitions ...................................................................................................... 11
3.2 Requirements Elicitation .............................................................................. 12
3.3 Requirements Analysis ................................................................................. 17
3.4 Requirements Validation.............................................................................. 21
3.5 Requirements Management ......................................................................... 22
3.6 Agile Requirements Engineering ................................................................. 22
3.7 Improving Requirements Engineering Processes ...................................... 23
3.8 Summary about Requirements Engineering .............................................. 28
4 Usability and usability testing .............................................................................. 31
4.1 Definitions of usability .................................................................................. 31
4.2 Metrics of usability ....................................................................................... 32
4.3 Usability testing ............................................................................................. 32
4.4 Usability testing methods ............................................................................. 33
4.5 Agile Usability testing ................................................................................... 38
4.6 Summary about usability and usability testing .......................................... 39
5 Research method ................................................................................................... 41
5.1 Case description ............................................................................................ 41
5.2 Research process ........................................................................................... 42
5.3 Data collection and analysis ......................................................................... 45
6 Suggested process model and good practices ..................................................... 48
6.1 The initial state of Software Development process .................................... 48
6.2 Customer segmentation ................................................................................ 49
6.3 Process model for Requirements Management .......................................... 51
6.4 Process model for Usability Testing ............................................................ 52
6.5 Techniques for involving the customers ..................................................... 53
6.6 Getting feedback from other stakeholders ................................................. 58
6.7 From Product Backlog item to a feature .................................................... 59
6.8 Implementing changes to current processes ............................................... 60
7 Lessons learned ..................................................................................................... 61
7.1 Survey............................................................................................................. 61
7.2 Selecting participants for the customer collaboration ............................... 65
7.3 Interviewing ................................................................................................... 66
7.4 Observing ....................................................................................................... 67
7.5 User testing .................................................................................................... 68
7.6 Analysing all the collected data as a whole ................................................. 69
7.7 Other implemented practices ....................................................................... 70
8 Discussion .............................................................................................................. 73
8.1 RQ1: Applying RE practices to a SaaS development ................................ 73
8.2 RQ2: Usability Testing practices during development .............................. 74
8.3 Limitations of the study ................................................................................ 75
9 Conclusions ............................................................................................................ 77
10 References .............................................................................................................. 79
Attachments ................................................................................................................... 84
3

1 Introduction
1.1 Motivation
The goal of this thesis is to develop and improve requirements engineering and usability
testing practices for a SaaS product that has multiple customer segments. Agile devel-
opment practices can be implemented quite easily "by the book" when there is either
one customer or clear segment of customers that have approximately similar needs. Sit-
uation gets much more complex when the customer base is complicated and contains a
lot of customers both from public and private sectors, different domains and different
sizes. It is not obvious what best practices would be for handling complex set of re-
quirements and what kind of metrics can be created for helping the prioritization. Also,
it is not self-evident, how usability testing practices should be arranged for existing and
complex product with multiple customer segments.
The case company for this thesis project is IMS Business Solutions Oy (Arter Oy since
April 2017) and the target product is IMS Software. The company lead has made a stra-
tegic decision to renew the current user interface of IMS Software. This thesis project is
going to be one of the starting points of this new user interface enhancement project.
Current IMS Software is quite complex and has multiple customer segments. The stra-
tegic decision of the company is to modernize the old user interface in order to become
more competitive in mobile/touchpad world and for being able to better respond to cus-
tomer needs with modern user interfaces. Currently the process for defining and priori-
tizing the development tasks and projects is not clearly defined. The motivation for this
project is to improve the user experience by deepening the customer involvement in the
development process. The key focus is to improve the workflow into being more agile
and Lean: involving the customers in the development process, re-defining requirements
engineering practices and establishing usability testing practices. The challenge for re-
quirements elicitation and management is that the product is used by many different
customers varying both in size and domain. The target product is sold as SaaS; there-
fore, all the customers get the changes when a new version of the software is released
and updated.
The writer of this thesis has been working in the target company for years, so she has a
good preunderstanding about the company, product and current processes. This thesis
project is great opportunity to combine new skills and information gained during the
master studies in Aalto University with prior knowledge about the target company.
4
1.2 Research problem and questions

As the target product has multiple customer segments and the current state of require-
ments engineering and usability testing practices could be better, the first step was to re-
define and improve these. The research problem for this thesis is:

How requirements can be defined and managed in a SaaS product that has multiple
customer segments?
The research problem is divided into two separate research questions. First one is relat-
ed to requirements engineering practices and second one for ensuring usability point-of-
view in the continuous development.
Research question 1
How to apply requirements engineering practices during agile SaaS development to
satisfy multiple customer segments?
Research question 2
How to implement usability testing practices during agile SaaS development to ensure
customer satisfaction?
These research questions are the crucial elements for improving the software develop-
ment practices of the target company. They are interesting as the customer group is ver-
satile and creates complexity when trying to do customer collaboration during the soft-
ware development. The target product is developed continuously, so improving these
practices has significance for the company and the product development.

1.3 Scope and focus of the thesis

The scope of this thesis is to focus on implementing new practices in agile workflow.
Those practices are based on requirements engineering and usability testing literature.
IMS Software is the target product that is offered to customers both as SaaS and as
software licence and it is being developed continuously. This thesis focuses on improv-
ing work practices in the development of this particular product. The target company
has also another product (ARC) that is being developed by another team so it is good to
keep in mind that it would be beneficial to create practices that could be implemented
also in the development of that product as well. The literature review is done by focus-
ing on practices that could be useful for agile SaaS development with multiple customer
segments.
5
1.4 Structure of the thesis

This thesis starts with the introduction chapter. The literature review is divided into
three main chapters – agile methodology, requirements engineering and usability and
usability testing. Agile methodology is a brief introduction to agile and Lean practices
and following two chapters introduces the key elements and most common work prac-
tices of requirements engineering and usability. Chapter five – research framework in-
troduces the target company and the software product. Research process and data analy-
sis methods are also explained in that chapter. Chapters six and seven focus on the em-
pirical study. Chapter six starts with current state analysis and customer segmentation.
Then there are suggestions for process models of how requirements engineering and
usability testing practices should be done. Then there is a list of suggested methods that
are advised to try out for ensuring customer collaboration. The end of the chapter con-
tains suggestions for the resourcing the personnel and how the new practices should be
implemented to the current practices. Chapter seven describes the lessons learnt from
trying out these methods and work practices. Discussion chapter (8) answers the re-
search questions and describes the limitations of this study. Conclusions chapter (9)
contains most important conclusions based on the results of this study. It also contains
suggestions for the future both for further study and for the company.

The empirical study was conducted by using insider action research method. The sug-
gested and tested work practices in the empirical study were selected based on the litera-
ture review. Figure 1 shows how the research questions relate to the contents of this
thesis. It focuses only on showing how the research questions relate to the literature
review and the suggestions-part of the empirical study. Also, the lessons learnt chapter
(7) is based on the research questions, but it is not shown here as it is follow-up to the
suggestions –chapter.
6

Figure 1. The relationship between research questions and structure of the thesis.