entire may 2010 issue

20 Pages·2010·3.06 MB·English

TE TER
We’re
back!
E s s e n t i a l f o r er s
May/June 2010 £ 4 / ¤ 5 v2.0 number 3
Where next
for testing
standards?
Including articles by:
Helen Davidson IDBS John Kent Simply Testing Graham Thomas Badgerscroft
Isabel Evans Testing Solutions Group Michiel van der Voort BCS Geoff Quentin

TE TER From the editor
Where next for testing standards?
What do testers want from testing need detail: saying your nominee was good
standards? Usability. is not enough. Please explain what was
achieved and how.
Academic prowess and professional rigour
are essential and admired, but justified only if If you have delivered something really special
their end product can be used to advantage for your customer, ask them to nominate you.
Where next
by a sufficient number of testers.
for testing In the next issue: testing across
The way forward is to make standards that boundaries
standards?
more testers can use.
Good testing is objective. Therefore it is not
The Professional Tester Reader's Award: affected by differences unconnected with
help fellow testers find excellence testing, for example those of personal origin,
culture or ability.
People that do excellent work to help others
improve software quality deserve recognition, But testing is not yet objective enough, so
and PT's readers are ideally placed to these things do affect it. We need ways to
Contact identify them. So we are inaugurating the work together despite our differences, to help
Professional Tester Reader's Award. us achieve our objectives in both the short
Editor (better testing) and long (objective testing
Edward Bishop Yes that apostrophe is in the right place! unaffected by irrelevant boundaries) terms. In
[email protected]
Winners will be chosen by individual readers. the next issue we'll learn about some ways.
Managing Director If your testing has been served well please
Niels Valkering
tell us about it. Any individual or organization Edward Bishop
[email protected]
can be nominated. To confer an award we Editor
Art Director
Christiaan van Heest
[email protected]
Sales
Rikkert van Erp IN THIS ISSUE
HakeemSerbouti
Where next for testing standards?
[email protected]
4 ISO 9001 7.x
Contributors to this issue: Helen Davidson explores applying a quality standard to testing-as-a-service
Helen Davidson
John Kent
Graham Thomas 8 EMT
Isabel Evans
John Kent proposes a new kind of testing standard
Rikkert van Erp
Michiel van der Voort
Geoff Quentin 11BS 7925-3?
Publisher The work towards a standard for non-functional testing, explained by
Jerome H. Mol Graham Thomas and Isabel Evans
[email protected]
Subscriptions 14 DO-178C + UML
[email protected]
Rikkert van Erp reports from iqnite Germany 2010
Professional Tester is published Interview
by Test Publishing Ltd. 15 International relationships step up a level
Michiel van der Voort answers our questions on the demise of ISEB Practitioner
We aim to promote editorial independence
and free debate: views expressed by
contributors are not necessarily those of the Series
editor nor of the proprietors. 17 CTP2
©Test Publishing Ltd 2010.
The second instalment of Geoff Quentin's Consolidated Testing Process
All rights reserved. No part of this
publication may be reproduced in any form
without prior written permission. ON THE WEB
“Professional Tester” is a trademark of Test
McAfeed: a tester's view Test Library Rude Coarse Analysis readers' letters
Publishing Ltd.
Visit professionaltester.com for the latest news and commentary
3
PT - May 2010 - professionaltester.com
Where next for testing standards?
ISO 9001 7.x
by Helen Davidson
Applying generic “quality” standards to
testing can appear complicated,
perhaps because it is itself concerned
with quality. But understanding what
those standards mean to testing might
lead to valuable improvement
it as such clearly think so. So does
Ashfaque Ahmed, author of the book
Software Testing as a Service (ISBN-13:
978-1420099560). Ivan Ericsson of SQS
advocated centralizing testing within an
organization in the March 2010 issue of
Professional Tester.
I asked various software people who
expressed quite strong but differing
views. I'd like to be able to say that the
testers among them agreed with one
another, but they did not. I believe their
opinion related to their organization's
view and treatment of testing: testers
who felt the test function within which
they worked was able to act
independently found it easier to view it
as a service. Perhaps then making
testing more independent makes it
resemble a service more, or perhaps
Helen Davidson IDBS, which produces software for treating it as a service can help to make
research and development organizations, it more independent. Many sources, for
explores ISO 9001 in
is an ISO 9001 certified company. As a example testing syllabuses such as
the context of testing test team lead there I have experienced ISTQB Certified Tester Foundation Level,
several certification audits and many identify test independence as desirable.
customer audits. During all of them, when And outsourced testing is a service by
talking to me, the auditors referred to definition.
clause 7.3.6 Conduct design and
development validations. But I think more I am lucky that independent testing is
of 9001 than this applies to software valued in my current project and that I
testing: if testing is a service, the whole of am in direct ownership of the approach
section 7 Product realization becomes and activities. I definitely feel that the
relevant to it. test teams are independent and provide
a service. So, would applying section 7
Is software testing a service? help us to define or improve our test
The many consultancies currently selling process and the deliverables required for
4
PT - May 2010 - professionaltester.com
Where next for testing standards?

ISO 9001 – TickIT quality plan Test strategy

Customer specified and implied quality Test objectives
requirements
Quality requirement for the project
Key features of the product and/or Lists of project and product risks or a
service that are likely to impact upon reference to the risk register and the
quality prioritisation of requirements, objectives
or work items
Specific risks Lists of project and product risks or a
reference to the risk register
References to quality system References to test (or other) procedures
procedures to be used to be used, eg test case review
Methods, tools etc List of test tools (eg test management
tool) and/or test methodologies (eg test
design techniques) or organizational test policy. The reporting
frequency and mechanism must be
Relevant statutory and regulatory For example US Food and Drug defined in the 9001 “Quality Management
requirements Administration guideline 21 CFR Part 11
System”. Our test team collaborates with
(varies with sector)
its customers on the test strategy using
Verification and validation strategy Test specification and test case review Microsoft SharePoint: this makes providing
evidence of communication to auditors
Exit criteria for procedures Entry and exit criteria for phases of
easy.
software testing
7.3 Design and development
Deviations and in new initiatives to Processes that are not being followed
standard processes and details of why; details of initiatives It's important to keep sight of the fact that
being trialled in the current context this is about
planning, designing and developing the
Table 1: Quality planning elements in a test strategy
testing, not the software being produced.
7.3 requires that inputs are defined and
it? Or would it just generate unnecessary 7.2 Customer-related processes outputs provided in a form that enables
paperwork? To try and find out, I studied To move toward compliance first the them to be verified against the design. Any
the section in depth, attended two customer must be identified. The test test process that documents its strategy
relevant training courses, then set about function can be considered as providing a and plans effectively should achieve this.
comparing our process with section 7 service to developers, business analysts,
looking for gaps. Where I found them, I product owners or the software buyer The subclauses go on to require that
considered what we would need to do to depending upon the current objective of systematic reviews, verification and
achieve compliance. testing, so the customer varies with sector, validation of the design and development
organization, context, testing phase and are performed. It may be argued that for a
Clause 7.1 Planning of product testing activity. testing service the reviews can be of its
realization outputs, eg test documentation. If so, this
TickIT, a guide to interpreting the Perhaps the best approach is to consider activity can also achieve verification of
requirements of 9001 for the software the information produced by testing. these against the input requirements. If
industry (see http://tickit.org), addresses Whoever receives and uses that, whether the customer as identified for clause 7.2
this with its “quality plan”, a description of to correct defects or inform decision takes part, these reviews might be able to
how the product or service provided is to making, is the customer. For compliance, achieve validation of the testing too.
be developed focusing on project-specific the test service must identify and review
decisions: methods, tools and techniques. the customer's requirements and Finally, 7.3 requires that changes to
When the service is software testing, this communicate with the customer. For a test design and development are recorded,
is what most testers call the test strategy. service that is achieved by reporting, so reviewed, verified and validated. It might
Table 1 compares the quality plan with the the reporting objectives must be defined, be addressed using configuration
test strategy standard we use at IDBS. in either or both of the project test strategy management of the inputs and outputs,
5
PT - May 2010 - professionaltester.com
TE TER
E s s e n t i a l f o r s o f t w a r e t e s t e r s
Where next for testing standards?

but a better interpretation is probably that compliance with the described processes testing standards is to consider what they
it indicates the use of a defined and allow correction of any deviations. are trying to achieve and how it applies to
mechanism for test process improvement. what we as testers understand to be good
7.6 Control of monitoring and testing practice: to help us understand why
7.4 Purchasing measuring devices our process is as it is, buy into it fully, and
This is a hard one. It requires evidence of This seems to refer to the methods used improve it continually.
a formal selection, evaluation and for the monitoring and measurement of
verification process used to choose the testing required by 7.5. Calibrating these As I have tried to show in this article, a
test tools being used. If that took place and safeguarding them from incorrect sequential test process seems to align
some time ago, or in fact the process used adjustments, damage or deterioration, or reasonably well with both the letter and
was not sufficiently systematic, the only even defining what that means, would be a spirit of 9001 section 7. What about testing
solution might be to re-specify, re-evaluate challenge: but it may not be necessary for as it is done to support an “agile”
and re-verify the tools, plus alternatives, to compliance because the clause states that development approach? Perhaps
justify (or change) past choices. it need be done only “where necessary to considering that in the same way would
ensure valid results”. A periodical help to improve it too
The clause could apply also to the examination of the reporting mechanisms,
purchase of consultancy and outsourced perhaps including test runs with invented
testing services. input data, might be considered sufficient.
7.5 Production and service provision Can test organizations achieve
To meet the requirements of this it is compliance with 9001 and is it
necessary to provide evidence that the worthwhile?
test service is provided under controlled It is possible: the many certified
conditions. The definition of these in 7.5 outsourced testing service providers have
seems to align well with the test policies, proved that. So do their auditors follow
strategies and plans, monitoring and similar reasoning to mine about section 7?
measurement of test effectiveness, and Could they audit an internal test provision
configuration management mechanisms in the same way? It may be as well for all
typically used by a test organization. In organizations that test software to be
that case, the required verification of these ready in case they do.
can be achieved by internal audit. The
requirement to exercise care with But compliance does not mean the test
customer property is probably met by process is effective or the software
prevailing data protection and other products are good. I think the best way to
governance standards. derive benefit from 9001 and other non-
It also requires that the processes are Helen Davidson is test team lead at IDBS (http://idbs.com) which provides advanced
verified and in most organizations software solutions to research and development organizations, including those working
monitoring of this adherence is done in pharmaceuticals, agrochemicals, animal health, cosmetics, petrochemicals, forensics,
through internal audits that demonstrate manufacturing, academic research and government, worldwide.
Where next for testing standards?
EMT
from the requirements for a collaboration
platform with two levels of user privileges:
SI1: Only an administrator can
delete a user
by John Kent
This might cause a tester to derive two
To define the things used to test, describe the
test conditions:
relationships between them
TC1: An administrator can delete a user
TC2: A user cannot delete another user
The things we use to test are poorly
defined. Glossaries and vocabularies And if there were more privilege levels,
define terminology but not the relationships there would be more TCs. So we can
between the entities defined. Standards assert that
and syllabuses imply some of those A specification item can give rise to many
relationships but do not define them. test conditions
IEEE 829 Standard for Software Test
Documentation includes a diagram In data modelling terms, there is a one-to-
showing relationships between the many relationship between the entities.
documents it defines, but not with other
related entities. Other standards and However this example is too simple.
sources describe test process, methods Suppose another specification item exists:
and techniques, but I believe any
description that does not include SI50: A user attached to a current project
definitions of all the entities it involves and cannot be deleted
John Kent proposes an the relationship between them is
incomplete. That full definition – an entity This might create the test conditions:
addition to testing
model for testing – would make process,
standards methods and techniques easier to define TC53:A user attached to a current project
and understand and therefore to improve. cannot be deleted
TC54:A user not attached to a current
Developing the model project can be deleted
The model should describe all the possible
relationships, not be limited to ones based The real meaning of both of these is
on common ways of arranging test artifacts qualified by SI1, showing that the reverse
and deliverables. That would impair its relationship exists too:
usefulness as a tool for comparing and A test condition can be related to many
evolving different ways of working. To specification items
identify a relationship we will (i) cite or
envisage examples of it in practice, then This can be depicted as in figure 1.
(ii) formulate generic statements about it.
Regarding the definitions of the entities Test condition and test case
themselves, for simplicity in this illustration The reasoning for these two entities is
I have used definitions I assume to be in similar: a test condition gives rise to many
common use and/or those given in the test cases, but each of those test cases
glossaries and vocabularies familiar to often satisfies other test conditions as well.
testers. Using different definitions would
lead to different relationships; a complete So the relationship is again many-to-many.
entity model will require unambiguous
definitions of the terms it includes. Test case and test script
A test script can contain many test cases.
Specification item and test condition A test case can appear in many test
Consider the following specification item, scripts
8
PT - May 2010 - professionaltester.com
Where next for testing standards?
Test script and test execution schedule running only one test execution schedule
A test execution schedule can contain only once (iii)
many test scripts. A test script can be
included in many test execution schedules But if a log can be a compilation of results
from multiple schedules and runs:
Test execution schedule and test A test results log can contain results from
results log running many test execution schedules
If each test execution schedule is unique, once each
in other words a different version of it is
created for each time the set of test If case (ii) above is true, depending on the
procedures it includes is to be run: nature of the log, the relationship is either:
A test execution schedule can be referred A test results log can contain results from
to by only one test results log (i) running many test execution schedules
once
But if a schedule uses relative dates, that or
is the same schedule can be run more A test results log can contain results from
than once (eg for repeated test cycles and running many test execution schedules
regression testing), then: each many times
A test execution schedule can be referred
to by many test results logs (ii) A third possibility is to maintain a log
corresponding to each schedule and
The reverse relationship depends on the append new results to it each time that
nature of a test results log. If case (i) schedule is run. In that case:
above is true, and if a new log is created A test results log can contain results from
each time a schedule is run: only one test execution schedule run many
A test results log can contain results from times
Figure 1: a specification item can give rise to many test conditions and a test condition can be related to many specification items
Specification item Test condition
Figure 2: part of an entity model for testing
Specification item Test condition Test case Test script Test execution Test results log
schedule
Figure 3: test execution without scripting
Specification item Test condition Test execution Test results log
schedule
Figure 4: exploratory testing
Specification item Incident report Test script
Figure 5: HP Quality Center
Specification item Test condition Test script Test execution Test results
schedule
Test step
9
PT - May 2010 - professionaltester.com
Where next for testing standards?
Figure 2 depicts all the relationships “concurrent test design, test execution, This is depicted in figure 4.
identified so far, assuming cases (i) and test logging and learning”. This could be
(iii) above are true. taken to mean there are no test conditions The way HP Quality Center works can be
or script and therefore no test schedule depicted as an entity model (figure 5).
This model is far from complete. Other prior to execution. However specification
possibilities, for example unscheduled items must exist, even if only in the mind of Using the model
execution of tests (and therefore creation the tester: otherwise the activity is just I suggest that standards dealing with testing
of results logs), need to be analysed and exploration and not testing at all. processes and terminology should include
added as relationships. Obvious entities to an entity model to complete definition of the
be added include test items, test plans and When an incident is found and reported it products and deliverables to which they
incident reports. is usual to include instructions on how to refer.
reproduce it, and these could be
Example variations of the model considered a test script. So: Even more importantly, I hope that testers
Suppose the person executing the tests An incident report refers to one test script will find that applying this analysis method
has good understanding of the SUT and and to the process they use helps them to gain
strong business knowledge. It can be A test script describes how to reproduce better understanding and definition of that
argued that he or she can execute test one incident process, and useful ideas for improving it
conditions directly and further analysis and
scripting is unnecessary (figure 3) and that John Kent is managing director of Simply Testing Ltd (http://simplytesting.com).
this is a way to improve coverage by
identifying and testing more test This article is heavily abridged and adapted from a paper presented at EuroSTAR 2008
conditions. (see http://www.eurostarconferences.com) available at
http://simplytesting.com/Downloads/ent.pdf which drew on discussions with members of
Exploratory testing is defined by some as the Software Testing Retreat to whom the author is indebted.
SIGIST Conference
Tuesday 29 th June 2010
21s t Birthday Celebration
Venue: Royal College of Obstetricians and Gynaecologists,
Regents Park, London, NW1 4RG
Our theme for June is “Testing Then and Now” - How Testing has
changed over our 21 years, in celebration of the BCS Software
Testing Specialist Group's 21st birthday. We will also be holding a
drink and nibbles social event after the conference closes at 17.00,
courtesy of Microsoft, which delegates are welcome to attend at no
extra charge.
This is a great conference for you to learn, improve your skills and
network with other organisations and testers.
Full day attendance is
£100 (excluding VAT) for BCS members
£110 (excluding VAT) for non members.
This includes lunch and handouts, as well as entrance to the tools
& services exhibition.
There is also a reduced rate for full time students.
For further information and the latest agenda please visit
www.sigist.org.uk.