Testing Techniques in Software Engineering: Second Pernambuco Summer School on Software Engineering, PSSE 2007, Recife, Brazil, December 3-7, 2007, Revised Lectures

322 Pages·2010·3.76 MB·English

by  Patrícia Machado

Lecture Notes in Computer Science 6153
CommencedPublicationin1973
FoundingandFormerSeriesEditors:
GerhardGoos,JurisHartmanis,andJanvanLeeuwen
EditorialBoard
DavidHutchison
LancasterUniversity,UK
TakeoKanade
CarnegieMellonUniversity,Pittsburgh,PA,USA
JosefKittler
UniversityofSurrey,Guildford,UK
JonM.Kleinberg
CornellUniversity,Ithaca,NY,USA
AlfredKobsa
UniversityofCalifornia,Irvine,CA,USA
FriedemannMattern
ETHZurich,Switzerland
JohnC.Mitchell
StanfordUniversity,CA,USA
MoniNaor
WeizmannInstituteofScience,Rehovot,Israel
OscarNierstrasz
UniversityofBern,Switzerland
C.PanduRangan
IndianInstituteofTechnology,Madras,India
BernhardSteffen
TUDortmundUniversity,Germany
MadhuSudan
MicrosoftResearch,Cambridge,MA,USA
DemetriTerzopoulos
UniversityofCalifornia,LosAngeles,CA,USA
DougTygar
UniversityofCalifornia,Berkeley,CA,USA
GerhardWeikum
Max-PlanckInstituteofComputerScience,Saarbruecken,Germany
Paulo Borba Ana Cavalcanti
Augusto Sampaio Jim Woodcock (Eds.)
Testing Techniques
in Software Engineering
Second Pernambuco Summer School
on Software Engineering, PSSE 2007
Recife, Brazil, December 3-7, 2007
Revised Lectures
1 3
VolumeEditors
PauloBorba
UniversidadeFederaldePernambuco
CentrodeInformática
CEP50732-970,Recife,PE,Brazil
E-mail:[email protected]
AnaCavalcanti
UniversityofYork
DepartmentofComputerScience
Heslington,YorkYO105DD,UK
E-mail:[email protected]
AugustoSampaio
UniversidadeFederaldePernambuco
CentrodeInformática
CEP50732-970,Recife,PE,Brazil
E-mail:[email protected]
JimWoodcock
UniversityofYork
DepartmentofComputerScience
Heslington,YorkYO105DD,UK
E-mail:[email protected]
LibraryofCongressControlNumber:2010929777
CRSubjectClassification(1998):D.2.4,D.2,D.1,F.3,K.6.3,F.4.1
LNCSSublibrary:SL2–ProgrammingandSoftwareEngineering
ISSN 0302-9743
ISBN-10 3-642-14334-2SpringerBerlinHeidelbergNewYork
ISBN-13 978-3-642-14334-2SpringerBerlinHeidelbergNewYork
Thisworkissubjecttocopyright.Allrightsarereserved,whetherthewholeorpartofthematerialis
concerned,specificallytherightsoftranslation,reprinting,re-useofillustrations,recitation,broadcasting,
reproductiononmicrofilmsorinanyotherway,andstorageindatabanks.Duplicationofthispublication
orpartsthereofispermittedonlyundertheprovisionsoftheGermanCopyrightLawofSeptember9,1965,
initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer.Violationsareliable
toprosecutionundertheGermanCopyrightLaw.
springer.com
©Springer-VerlagBerlinHeidelberg2010
PrintedinGermany
Typesetting:Camera-readybyauthor,dataconversionbyScientificPublishingServices,Chennai,India
Printedonacid-freepaper 06/3180
Preface
The Pernambuco School on Software Engineering (PSSE) 2007 was the second
in a series of events devoted to the study of advanced computer science and to
the promotion of international scientific collaboration.The main theme in 2007
was testing. Testing is nowadays a key activity for assuring software quality.
Thesummerschoolandits proceedingswereintendedtogiveadetailedtutorial
introduction to the scientific basis of this activity and its state of the art.
Theseproceedingsrecordthecontributionsfromtheinvitedlecturers.Eachof
thechaptersistheresultofathoroughrevisionoftheinitialnotesprovidedtothe
participants of the school. The revision was inspired by the synergy generated
by the opportunity for the lecturers to present and discuss their work among
themselves and with the school’s attendees. The editors have tried to produce
a coherentview of the topic by harmonizing these contributions,smoothing out
differences in notation and approach, and providing links between the lectures.
We apologize to the authors for any errors introduced by our extensive editing.
Althoughthechaptersarelinkedinseveralways,eachoneis sufficientlyself-
contained to be read in isolation. Nevertheless, Chap. 1 should be read first by
those interested in an introduction to testing.
Chapter 1 introduces the terminology adopted in this book. It also provides an
overview of the testing process, and of the types (functional, structural, and so
on)anddimensions(unit,integration,andsoon)ofthetestingactivity.Themain
strategies employed in the central activity of test selection are also discussed.
Mostofthe materialpresentedinthisintroductorychapterisaddressedinmore
depth in the following chapters.
Chapter 2 givesanoverviewofthe foundationsandpracticeoftesting.Itcovers
functional, structural, and fault-based testing, and discusses automation exten-
sively.Evaluationandcomparisonarebasedonexperimentation,withemphasis
on mutation testing.
Chapter 3 discussestest-case generationandselectionfor reactivesystems.The
focus is on model-based approaches. A particular tool, Target, and a case
studyusingmobile-phoneapplicationsareusedforextensiveillustrationoftech-
niques,strategies,andalgorithms.Modellingisanexpensiveactivityinthiscon-
text, and this chapter also discusses the automatic generation of models from
requirements.
Chapter 4 discusses the main challenges of testing software product lines. It
presents several techniques and an overall process for testing, in a productive
way,products that have related functionalities but specific behavior differences.
Thischapterconsiderstestingandvalidationnotonlyofcodebutotherartifacts
as well.
VI Preface
Chapter 5 describes how to design, implement, and test software using the
methodology of ParameterizedUnit Testing. This is supported by the tool Pex,
an automatedtest input generator,which leveragesdynamic symbolic execution
to testing whether the software under test agrees with the specification. Pex
produces a small test suite with high code coverage from Parameterized Unit
Tests.The basicconceptsareintroducedandthe techniquesareillustratedwith
some examples. Deterministic, single-threaded applications are assumed.
Chapter 6 summarizes the experience of a leading tool vendor and developer.
It addresses the issues governingthe design and use of both static and dynamic
testing tools. It argues that a well-designedmodern toolrelies on anunderlying
mathematical theory: there is now a convergence between formal methods and
principled testing.
Chapter 7 presentsagenericframeworkfordevelopingtestingmethodsbasedon
formal specifications, and its specialization to several formal approaches:Finite
State Machine, Algebraic Specifications, Input-Output Transition Systems, and
TransitionSystemswithPriorities.Assumingsometestabilityhypothesisonthe
systemundertest,anotionofexhaustive test suite isderivedfromthesemantics
of the formalnotation and fromthe definition of correctimplementation. Then,
a finite test suite can be selected from the exhaustive one via some selection
hypotheses.
Chapter 8 revisits fault-based and mutation testing. It gives a foundational ac-
countofthistechniqueusingarelationalmodelofprogrammingbasedonrefine-
ment,namely,thatofHoareandHe’sUnifyingTheoriesofProgramming(UTP).
Thetheorysuggestsandjustifiesnoveltest-generationtechniques,whicharealso
discussed. Tool support for the presented techniques is considered.
We aregratefulto the members ofthe OrganizingCommittee,who workedvery
hard to provide an enjoyable experience for all of us. Without the support of
our sponsors, PSSE 2007 could not have been a reality. Their recognition of
the importance of this event for the software engineering community in Latin
America is greatly appreciated.We wouldalso like to thank all the lecturers for
their invaluable technical and scientific contribution, and for their commitment
to the event; the effort of all authors is greatly appreciated. Finally, we are
grateful to all the participants of the school. They are the main focus of the
whole event.
March 2010 Paulo Borba
Ana Cavalcanti
Augusto Sampaio
Jim Woodcock
Organization
PSSE 2007 was organized by the Centro de Informática, Universidade Fed-
eral de Pernambuco (CIn/UFPE), Brazil, in cooperation with the University
of York, UK.
Executive Committee
Paulo Borba CIn/UFPE Managing Director and Guest Editor
Ana Cavalcanti University of York
Augusto Sampaio CIn/UFPE
Jim Woodcock University of York
Sponsoring Institutions
Formal Methods Europe
Sociedade Brasileira de Computação, Brazil
United Nations University, Macau
Universidade Federal de Pernambuco (CIn/UFPE), Brazil
University of York, UK
VIII Organization
Acknowledgements
AuriVincenzi,MárcioDelamaro,ErikaHöhn,andJoséCarlosMaldonado would
like to thank the BrazilianFunding Agencies – CNPq, FAPESP and CAPES —
and the QualiPSo Project (IST-FP6-IP-034763)for their partial support of the
researchthey report in Chap. 2.
Patricia Machado and Augusto Sampaio would like to emphasize that most of
theirchaptercoversresultsachievedfromaresearchcooperationinsoftwaretest-
ing between Motorola Inc., CIn-UFPE, and UFCG. They would like to thank
the entire group for all the support, effort, criticisms, and suggestions through-
out this cooperation. Particularly, their chapter is based on joint papers with
Gustavo Cabral, Emanuela Cartaxo, Sidney Nogueira, Alexandre Mota, Dante
Torres, Wilkerson Andrade, Laisa Nascimento, and Francisco Oliveira Neto.
John McGregor would like to thank Kyungsoo Im and Tacksoo Im for their
work on implementations and John Hunt for the implementations from his
dissertation.
Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte would like to
thank their past interns and visiting researchers Thorsten Schuett, Christoph
Csallner, Tao Xie, Saswat Anand, Dries Vanoverberghe, Anne Clark, Soonho
Kong, Kiran Lakhotia, Katia Nepomnyashchaya, and Suresh Thummalapenta
for their work and experiments to improve Pex, Nikolaj Bjørner and Leonardo
de Moura for their work on the constraint solver Z3, the developers and testers
of .NET components and Visual Studio within Microsoft for their advice, and
all users of Pex for giving feedback.
Marie-Claude Gaudel’s text is extracted from or inspired by previous articles
co-authoredby the author.She wouldparticularlylike to thank PascaleLe Gall
for Sect. 2, Richard Lassaigne, Michel de Rougemont, and Frédéric Magniez
for Sect. 3, Perry James and Grégory Lestiennes for Sects. 4 and 5, and Ana
Cavalcanti for new exciting work on testing CSP refinements.
BernhardK.Aichernig’s workwascarriedoutaspartoftheEU-fundedresearch
projectinFramework6:IST-33826CREDO(Modelingandanalysisofevolution-
ary structures for distributed services). His Theorem 12 was contributed by He
Jifeng,thenatUNU-IIST.Severalpeoplecontributedtotheimplementationsof
thetheoriesdiscussedinSect.6ofhischapter.TheOCLtest-casegeneratorwas
implemented by Percy Antonio Pari Salas during his fellowship at UNU-IIST.
The Spec#test-casegeneratorisanideaofWillibaldKrenn,TUGraz,andwas
realized in the EU project in Framework 7: ICT-216679 MOGENTES (Model-
basedGenerationofTestsforDependableEmbeddedSystems).ThefirstLOTOS
mutation-testingcasestudywascarriedoutbyCarloCorralesDelgadoduringhis
fellowship at UNU-IIST. The more recent achievements in the mutation testing
of protocols are the work of Martin Weiglhofer, TU Graz.
Table of Contents
Software Testing: An Overview .................................... 1
Patr´ıcia Machado, Auri Vincenzi, and Jos´e Carlos Maldonado
Functional, Controland Data Flow, and Mutation Testing: Theory and
Practice ........................................................ 18
Auri Vincenzi, Ma´rcio Delamaro, Erika H¨ohn, and
Jos´e Carlos Maldonado
Automatic Test-Case Generation................................... 59
Patr´ıcia Machado and Augusto Sampaio
Testing a Software Product Line ................................... 104
John D. McGregor
ParameterizedUnit Testing with Pex: Tutorial....................... 141
Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte
Software Tool Issues.............................................. 203
Michael Hennell
Software Testing Based on Formal Specification...................... 215
Marie-Claude Gaudel
A Systematic Introduction to Mutation Testing in Unifying Theories of
Programming.................................................... 243
Bernhard K. Aichernig
Author Index.................................................. 313
Software Testing: An Overview
PatríciaMachado1,AuriVincenzi3,andJoséCarlosMaldonado2
1 UniversidadeFederaldeCampinaGrande,Brazil
[email protected]
2 UniversidadedeSãoPaulo,Brazil
[email protected]
3 Universidade Federal de Goiás, Brazil
[email protected]
The main goal of this chapter is to introduce common terminology and concepts on
softwaretestingthatisassumedasbackgroundinthisbook.Thechapteralsopresents
themultidimensionalnatureofsoftwaretesting,showingitsdifferentvariantsandlevels
ofapplication.
After a brief introduction, Section 2 presents a set of basic definitions used in the
remainingofthisbook.Section3givesanoverviewoftheessentialactivitiesanddoc-
uments involved in most test processes. Section 4 discusses the kinds of properties
we may want to test, including functional, non-functional, and structural properties.
InSection5,wediscussthevariousdimensionsofsoftwaretesting,coveringunit,in-
tegration, system, and acceptance testing. Section 6 highlights that differentdomains
havedemandedeffortfromtheresearchcommunitytoworkontailoredstrategies;we
discuss object-oriented, component-based,product-line, and reactive-systems testing.
Testselectionisamainactivityofatestprocess,andwediscussthemainstrategiesin
Section7.WeconcludethisintroductioninSection8withsomefinalconsiderations.
1 Introduction
Inrecentyearstheinterestandimportanceofsoftwaretestinghavegrown,mainlydue
totherisingdemandforhighersoftwarequality.Shulletal.[298]presentadiscussion
andaninterestingsynthesisinthecurrentknowledgeofsoftwaredefectreduction.One
pointtheycommentonconcernsthesoftwaremodulesthatmostcontributetodefects.
Theywarnthat,duringdevelopment,virtuallynomodulesaredefect-freewhenimple-
mentedandabout40%ofthemodulesmaybedefect-freeaftertheirrelease.Therefore,
aswaspointedoutbyBoehmandBasili[47],itisalmostimpossibletodeliverasoft-
wareproductfreeofdefects.
Moreover,itisimportanttoobservethatthelaterafaultisdetected,thegreaterthe
cost of its correction (see Figure 1). In 1987, Boehm [48] evaluated the relative cost
of a defect correctionagainst the developmentphases it detected and concludedthat,
fromrequirementstomaintenance,thecost-escalationfactorrangesfrom1to100(Fig-
ure1(a)),where1meanstherelativecostintherequirementphaseand100referstothe
costinthemaintenancephase.Morerecently,BoehmandBasili[47]providednewand
not so dramatic informationthat the cost-escalation factor for small, noncritical soft-
waresystemsismorelikelytobe5:1than100:1(Figure1(b)).However,evenafactor
P.Borbaetal.(Eds.):PSSE2007,LNCS6153,pp.1–17,2010.
(cid:2)c Springer-VerlagBerlinHeidelberg2010