A Theoretical Framework for Understanding Mutation-Based Testing Methods

0.88 MB·

by  Donghwan Shin

A Theoretical Framework for Understanding
Mutation-Based Testing Methods
Donghwan Shin Doo-Hwan Bae
School of Computing School of Computing
KAIST KAIST
Daejeon, Republic of Korea Daejeon, Republic of Korea
Email: [email protected] Email: [email protected]
6 Abstract—In the field of mutation analysis, mutation is the studies in software testing [12]–[14], and defined a formal
1 systematic generation of mutated programs (i.e., mutants) from modelfortesting,calledatestingsystem.Thisincludesfunda-
0 an original program. The concept of mutation has been widely
mentaltestingfactors(i.e.,programs,specifications,tests)and
2 applied to various testing problems, including test set selection,
their formal relationships. Staats et al. [15] recently revisited
faultlocalization,andprogramrepair.However,surprisinglylittle
n focus has been given to the theoretical foundation of mutation- thetestingsystembyintroducingafourthtestingfactor,atest
a
based testing methods, making it difficult to understand, orga- oracle, which had previously been implicitly considered. This
J
nize, and describe various mutation-based testing methods. testing system delivers a holistic view of the testing factors,
5
This paper aims to consider a theoretical framework for
and serves as a guide for discussions in studies such as [16]–
2
understanding mutation-based testing methods. While there is
[19].However,thisisincongruentwithmutation-basedtesting
asolid testingframeworkfor generaltesting,this isincongruent
] with mutation-based testing methods, because it focuses on the methods, because it focuses on the correctness of a program
E
correctnessofaprogramforatest,whiletheessenceofmutation- foratest,whiletheessenceofmutation-basedtestingconcerns
S
basedtestingconcernsthedifferencesbetweenprograms(includ- the differences between programs (including mutants) for a
.
s ing mutants) for a test. test.
c In this paper, we begin the construction of our framework
For example, consider the recent mutation-based fault lo-
[ by defining a novel testing factor, called a test differentiator, to
calization studies of Papadakis and Traon [6] and Seokhyeon
transformtheparadigmoftestingfromthenotionofcorrectness
1
to the notion of difference. We formally define behavioral et al. [7]. In these studies, in order to find the exact locations
v
differences of programs for a set of tests as a mathematical of faults in a program (i.e., the original program with regard
6
6 vector,calledad-vector.Weexplorethemulti-dimensionalspace tomutants)itisessentialtoanalyzethebehavioraldifferences
represented by d-vectors, and provide a graphical model for
4 between mutants, the original program, and the specification
describingthespace.Basedonourframeworkandformalization,
6 (i.e., intended behavior), for a given set of tests. However,
we interpret existing mutation-based fault localization methods
0
and mutant set minimization as applications, and identify novel none of the existing theoretical frameworks can consistently
.
1 implications for future work. describe these various behavioral differences.
0 In this paper, we provide a solid theoretical framework
6 I. INTRODUCTION
for the notion of behavioral differences between programs
1
In the field of mutation analysis, mutation is the systematic in mutation-based testing. We begin the construction of our
:
v generationofmutatedprograms(i.e.,mutants)fromanoriginal framework by defining a novel testing factor, called a test
i
X program. DeMillo et al. [1] first proposed the notion of muta- differentiator, to transform the paradigm of testing from the
tion for measuring the quality of a set of tests using mutants notion of correctness to the notion of difference. Using the
r
a in the late 1970s. Mutation-based testing has been widely test differentiator, we formally define behavioral differences
studied with the aim of addressing various testing problems, between programs for a set of tests as a mathematical vector,
such as test set selection [2]–[4], robustness testing [5], fault called a d-vector. Based on the fact that a vector can be
localization [6], [7], and program repair [8]–[10]. However, regarded as representing a point in a multidimensional space,
surprisinglylittlefocushasbeengiventothetheoreticalfoun- wedefinethespaceofprogrambehaviorsforasetoftests,and
dation of mutation-based testing methods, making it difficult explore the theoretical properties of that space. We conclude
to understand, organize, and describe various mutation-based our theoretical framework with a graphical model to describe
testing methods. the space of behavioral differences. Based on our framework
This paper aims to consider a theoretical framework for and formalization, we interpret existing mutation-based fault
understanding mutation-based testing methods. A theoretical localization methods and mutant set minimization as applica-
framework is a well-formed model of the general entities that tions, and identify novel implications for future work.
areunderinvestigation,whichfacilitatesaclearunderstanding Theremainderofthispaperisstructuredasfollows.Section
ofthefundamentalsofcomplexproblems.Forexample,inthe II introduces some background material, and describes the
early 1980s Gourlay [11] organized the existing foundational scope of the mutation-based testing to be considered in this
paper. Section III describes our theoretical framework for Mutation-based testing differs from mutation analysis or
mutation-based testing, including the new testing factor, the mutation testing in terms of the way it utilizes mutants.
formal definition of behavioral differences, and our graphical Both mutation analysis and mutation testing are based on
model for behavioral differences. SectionIV presents applica- mutation adequacy, which means that they are based on the
tions of the proposed formal framework. Section V concludes differences between each of mutants and an original program.
the paper. However, mutation-based testing considers the differences not
only between each of mutants and an original program, but
II. MUTATION-BASEDTESTING also between each of mutants and a correct program (i.e.,
specification).Thedifferencesbetweenamutantandacorrect
Mutation analysis is a method for measuring the quality
program are particularly useful for fault localization and
of the set of tests, using artificially injected faults (mutants)
programrepair[7],[9],[10].Inthiscontext,acorrectprogram
generated from the original program using predefined rules
implies the intended behaviors of an original program for a
(mutation operators). If a mutant and the original program
given set of tests. While a correct program does not take the
returndifferentresultsforatest,thenthetestkillsthemutant.
form of an executable program with source codes, in practice
Mutation adequacy is satisfied when all of the generated
a human may play the role of the correct program, acting as
mutants are killed by the set of tests.
human oracle.
On the other hand, mutation testing provides a process As Offutt noted in [25], mutation applied to a program
for detecting faults in the original program. It starts with representsonlyoneinstanceofageneralclassofapplications.
the generation of mutants from the original program, and A general definition of mutation includes systematic changes
tests are generated with the aid of automatic test generation to the syntax or to objects developed from the syntax. While
methods. The generation of new tests and execution of live thispapermainlyfocusesonprogrammutation,theapplication
mutants are automatically repeated until the set of tests kills of results on this paper is not necessarily limited to programs,
all of the mutants. This loop is the key element of mutation but can be extended to general syntax or objects.
testing, which provides a set of tests satisfying the mutation
adequacy condition. After all of the mutants are killed, the III. THEORETICALFRAMEWORK
loop terminates and the original program is executed using To serve as a guide for discussions for both theoretical and
the resulting test set to detect faults. empirical studies, we focus on a concise framework rather
There are two main hypotheses justifying mutation testing: than a theory including axioms, theorems, and proofs used to
The Competent Programmer Hypothesis (CPH) [1] claims solve theoretical problems. In the following subsections, we
that the original program is made by competent programmers explain the foundational concepts, definitions, and examples
so the program has few simple faults. The coupling effect ofourtheoreticalframeworkforunderstandingmutation-based
hypothesis [20] states that complex faults are coupled to testingmethods.Theapplicationsofthetheoreticalframework
simple faults so the tests that kill simple mutants will detect are presented in Section IV.
a large percentage of complex faults.
A. Basic Terms and Notations
Several authors explored the theory of mutation in terms
Here, we will clarify the meanings of basic terms and no-
of test set selection for demonstrating the correctness of
tations used in this paper, including programs, specifications,
a program. Budd et al. [21] and Budd and Angluin [22]
tests, oracles, behaviors, and faults. We will then address the
presented the theoretical discussion on the test set selection
scope of mutation-based testing in comparison with mutation
problem in mutation testing. Morell [23] also discussed a
analysis and mutation testing.
theory of fault-based testing and considered the absence of
In terms of programs, specifications, tests, and oracles, we
prescribed faults in a program. On the other hand, many
willadoptthetestingsystemusedbyStaatsetal.[15],because
experimental studies have reported that tests satisfying the
it has a solid historical background, and provides an intuitive
mutation adequacy are effective at detecting faults [2], [3],
portrayal of the general testing process. The testing system is
[24]. While these theoretical and empirical studies provide
a collection (P, S, T, O, corr, corr ), where
solid background for mutation testing in terms of the correct- t
ness of a program, to the best of our knowledge, there is no • S is a set of specifications
theoreticalframeworksformutation-basedtestingbasedonthe • P is a set of programs
differences of programs. • T is a set of tests
By mutation-based testing we mean all mutation-based • O is a set of oracles
methods which attempt to solve various testing problems, not • corr ⊆P ×S
limited to fault detection problems. Mutation-based testing • corrt ⊆T ×P ×S
includes mutation-based fault detection, mutation-based fault A specification s ∈ S represents the true requirements of the
localization,andmutation-basedfaultremoval.Inotherwords, program p∈P. A test t∈T is a sequence of inputs accepted
if a testing method utilizes many mutants generated from an by some program. The correctness of a program is defined by
original program, then the method is regarded as one of the the predicate corr. For a program p ∈ P and a specification
mutation-based methods. s ∈ S, the predicate corr(p,s) implies that p is correct with
respect to s. Similarly, the predicate corr implies correctness
t
with respect to a test t ∈ T. In other words, corr (t,p,s)
t By definition, a test differentiator concisely represents
holds if and only if s holds for p when running t. The values
whether the behaviors of p ∈ P and p ∈ P are different
x y
ofcorr(p,s)andcorr (t,p,s)aretheoretical,andareusedto
t for t. Note that we make no attempt to incorporate any
describe the relationship between testing and correctness. By
specific definition of program differences in our framework.
definition,corr (t,p,s)isalwaystrueforalltestsifcorr(p,s)
t The specific definition of differences can only be decided in
is true. Because the predicate corr is theoretical, an oracle
t context. For example, while 0.3333 is different with 1/3 in
o∈O is defined as a predicate o⊆T ×P, which determines
the strict sense, 0.3333 will be regarded as the same as 1/3 in
the passing or failure of t for p in practice. In general, it
some cases. To keep things general, we consider a set of test
is assumed that o approximates corr , even though it is not
t differentiators D that includes all possible test differentiators.
perfect.
Having a test differentiator d as the fundamental testing
In terms of the behaviors of programs in testing, we adopt
factormakesitpossibletoformalizemanyimportantconcepts
the informal description given by Morell [23]: the behavior
inmutation-basedtestingmethods.Forexample,thenotionof
mayincludeanytestexecutionresults,forexampleitsoutput,
mutation adequacy, which is the essence of mutation-based
itsinternalvariables,itsexecutiontime,oritsspaceconsump-
testing, can be clearly and concisely formalized as follows:
tion. For example, the correctness of p for t refers to the
correctnessofthebehaviorofpfortheexecutionoft.Forthe ∀m∈M,∃t∈TS,d(t,p ,m). (1)
o
sake of simplicity, we assume that the behavior of a program
Inotherwords,allmutantsarekilledbyatleastonetestinthe
for a test is deterministic and independent of the previous
testsuiteTS ⊆T.Bytheformalizationofmutationadequacy
behavior. In the remainder of this paper, we will use the
in (1), it is shown that mutation adequacy is determined not
terms program and behavior of the program interchangeably.
only by p , m, and t, but also d. For example, there is a
Note that we do not take a black-box perspective, because the o
spectrum of mutation approaches from a strong mutation [1]
behavior includes internal variables and more.
to a weak mutation [26], depending on which d is used. In a
In terms of the faults of programs in testing, we say that t
strong mutation analysis, a test t kills a mutant m when the
detectsafaultofpwhenthebehaviorofpfortisinconsistent
outputofmdiffersfromtheoutputoftheoriginalprogramp
withs.Inotherwords,afaultisastaticdefectinpthatresults o
fort.Inaweakmutationanalysis,tkillsmwhentheinternal
inaninconsistencybetweentheintendedbehavior(i.e.,s)and
states of m and p are different for t. As a result, (1) implies
the behavior of p. o
that the holistic view of p , m, t, and d should be carefully
Throughout the remainder of this paper, p ∈ P refers to o
o
considered to meet a certain level of mutation adequacy.
the original program implemented to meet the specification
Consider an oracle o and a differentiator d, they are similar
s, and p ∈ P refers to the projection of s that represents
s
in terms of their role in testing; o implies the correctness of
the expected behaviors specified by s. While p is not a real
s
a program for a test, and d implies the differences between
program, this is not a serious assumption, because we only
programs for a test. In fact, for all o ∈ O, there are proper
require the behavior of p for a given set of tests. In practice,
s
d∈D and p ∈P where
ahumanmayplaytheroleofsorp ,actingasahumanoracle. s
s
The notation m∈M ⊆P refers to a mutant generated from
∀t∈T,∀p∈P,o(t,p)⇔¬d(t,p,p )
s
p ,andthesetofprogramsM referstoasetofmutants.Note
o
that po, ps, and m are general entities, and largely separated In other words, d can play the role of o with the aid of ps.
fromanyspecificssuchasprogramminglanguagesormutation For example, the correctness of a program p for a test t is
methods. written by not only o(p,t) but also d(t,p,p ). However, it is
s
clear that o cannot play the role of d in general. This means
B. Test Differentiator: A New Testing Factor
that d is more general than o. In the rest of this paper, d is
The notion of difference is an abstract concept. In order consistently used without o.
to concretize and formalize the notion of difference in our We should note that Staats et al. [15] formulated mutation
framework, we define a new testing factor, called a test adequacy as ∀m ∈ M,∃t ∈ TS,¬o(t,m). However, as they
differentiator, as follows: already mentioned, their formulation is inaccurate, because
general mutation adequacy does not include the term o(t,m),
Definition 1: A test differentiator d:T ×P ×P →{0,1}
which implies the correctness of m for t. Mutation adequacy
is a function,1 such that
is based on the differences between p and m for t, which is
o
(cid:40)
d(t,p ,p )= 1 (true), if px is different with py for t exactly captured by d(t,po,m).
x y
0 (false), otherwise
C. Behavioral Difference
for all tests t∈T and programs px,py ∈P. Beforeweformallydescribebehavioraldifferencesbetween
programs for a set of tests, we introduce a test vector, to
1Thisfunction-styledefinitionisreplaceablebyapredicate-styledefinition, formalize an ordered set of tests as follows:
suchasd⊆T ×P ×P.
D. Position: A New Interpretation of Behavioral Differences
A vector can be regarded as representing the position of a
point in a multi-dimensional space. For example, considering
a n-dimensional space, a vector v = (cid:104)v ,...,v (cid:105) represents
1 n
the point whose position in the i-th dimension (relative to the
origin of the space) is v , for i = {1,...,n}. In this way, we
i
can think of a d-vector as the representation of a position in a
multi-dimensional space. We introduce this new interpretation
Fig.1. Runningexamplefortests,programbehaviors,andd-vectors
of d-vectors as follows:
Definition 2: A test vector t = (cid:104)t ,t ,...,t (cid:105) ∈ Tn is a Definition 4: The position of a program px relative to an-
1 2 n
other program p in a multi-dimensional space corresponding
vector where t ∈T. r
i
to a set of tests t is
A test vector t ∈ Tn is the same as a test suite TS with
dt (p )=d(t,p ,p ),
sizen,exceptthattcontainsnumberedtests.Forexample,two pr x r x
tests tx,ty ∈ TS could form a test vector t = (t1,t2) ∈ T2. where d(t,pr,px) is the d-vector between px and pr for t,
In this paper, bold letters represent vector forms. with regard to d. This multi-dimensional space is called the
With the aid of d and t, we define d-vectors, which program space induced by (t,p ,d), where t corresponds to
r
formulates behavioral differences, as follows: the set of dimensions, p corresponds to the origin, and d
r
corresponds to the notion of differences between positions.
Definition 3: A d-vector d : Tn×P ×P → {0,1}n is an
n-dimensional vector, such that In other words, for all t ∈ t, the behavioral difference
between p and p for t is indicated by the position of
d(t,p ,p )=(cid:104)d(t ,p ,p ),...,d(t ,p ,p )(cid:105) r x
x y 1 x y n x y a program p relative to the origin p in the dimension t.
x r
for all t∈Tn, d∈D, and p ,p ∈P. Because d returns either 0 or 1, there are only two possible
x y
positions in each dimension: the same position as the origin
By definition, a d-vector d(t,p ,p ) represents the behav-
x y (i.e,. 0) and a different position from the origin (i.e., 1). It
ioral differences between p and p for all tests in t in vector
x y means that the semantics of a program p in the space of
form.Inotherwords,d(t,p ,p )effectivelyindicatesthetests x
x y (t,p ,d)isindicatedbythen-bitbinaryvectordt (p )where
t∈t for which the two programs px and py exhibit different n=r|t|. pr x
behaviors. For example, if d(t ,p ,p ) = 1 for some test
i x y The origin of a program space is important, because it
t ∈t,thismeansthatp andp aredifferentfortheparticular
i x y determines the meaning of positions in the program space. In
test t contained in the set of tests t.
i otherwords,theorigindeterminesthemeaningoftheprogram
In Figure 1, in a running example that we shall refer to
space. For example, if the correct program p is used for the
through the remainder of Section III, we present the program s
origin, then the position dt (p ) indicates how correct the
behaviorsofps,po,andmfort=(cid:104)t1,t2,t3,t4(cid:105)withd.Each program p is with regardpsto tx. On the other hand, if the
behavior of a program for a test is abstracted by a Greek x
original program p is used for the origin, and a mutant m
letter, and d determines a difference between behaviors by a o
generated from p is used for the target program p , then the
differencebetweenGreekletters.Ontheright-handside,there o x
position dt (m) indicates the killing of m with regard to t.
are d-vectors that represent the behavioral differences among po
Conceptually,thepositionofaprograminann-dimensional
the programs for the tests.
program space translates the program behavior as an n-bit
In the example, d(t,p ,p ) is equal to (cid:104)0,1,1,1(cid:105), because
s o binary string. Each bit represents the behavioral difference
d(t ,p ,p ) = 0, d(t ,p ,p ) = 1, d(t ,p ,p ) = 1, and
1 s o 2 s o 3 s o between the program and another program at the origin of
d(t ,p ,p )=1, respectively. Note that all of the behavioral
4 s o the space. In our example, the position of m relative to p
differencesamongp ,p ,andmarerepresentedbyd-vectors. o
s o is dt (m) = (cid:104)0,0,1,1(cid:105). This means that m is represented
A d-vector provides a quantitative difference by taking a po
by 0011 in the program space (t,p ,d). Such a concise
mathematical norm of vectors. In the example, for the d- o
representation makes it favorable to consider positions rather
vector d(t,p ,p ) = (cid:104)0,1,1,1(cid:105), the Manhattan norm2 gives
s o than d-vectors.
the quantitative difference as 0+1+1+1 = 3, in terms of
It is worthwhile to consider the norm of a position as well.
d. This means that the behavioral difference between p and
s Thenormofthepositionofp relativetop naturallyindicates
p is 3, quantitatively, in terms of the given t and d. This is x r
o thedistancefromp top intheprogramspace.Forexample,
written as ||d(t,p ,p )||=3. r x
s o the norm of the position of m relative to p is ||dt (m)||=2
o po
which means that the distance from p to m is 2. For an
2Ingeneral,consideringthebehaviorofaprogramforatestasastring,the o
quantitativedifferencebetweentwobehaviorsismeasurablebytheHamming arbitraryprogrampincludingmutants,||dt (p)||indicatesthe
ps
distance between two stringsfor thebehaviors. Interestingly,it isconsistent incorrectness of p with respect to t and d. For an arbitrary
with the Manhattan norm of a d-vector: the Hamming distance between the mutant m, ||dt (m)|| indicates the easiness of killing m with
behaviors of px and py for t is equivalent to the Manhattan norm of the po
d-vectord(t,px,py). respect to t and d.
Inourrunningexample,therearetwod-vectorsd(t,p ,p ) of the difference between two positions is related with the
s o
and d(t,p ,m). These indicate the two positions dt (p ) behavioral differences of programs in those positions. If two
s ps o
and dt (m) in the same program space, by Definition 4. In programsareindifferentpositions,thisguaranteesthatthetwo
ps
other words, we have a four-dimensional space, the origin is programs’behaviorsaredifferent.However,ifthepositionsare
p , and the two programs p and m are in (cid:104)0,1,1,1(cid:105) and not different, this does not guarantee that the two programs’
s o
(cid:104)0,1,1,0(cid:105), respectively. m is closer to the origin p than p behaviors are equal.
s o
which means that m is more correct than p . This shows
o F. Formal Relation on Positions
that mutation can generate partially correct mutants. This
idea is used to the foundational concept for mutation-based In the previous section, we discussed the relationship
fault localization and mutation-based program repair [7], [9], between the difference of positions and the difference of
[10]. Considering positions of many mutants makes it easy to behaviors. In this section, we introduce the formal relation
understand and discuss the mutation-based methods. We will on positions, called the deviance relation, as follows:
shows the specific application in Section IV-A.
Definition 5: For a program space defined by (t,p ,d),
r
E. Different Positions and Different Behaviors the position dtpr(py) is transitively deviant from the position
dt (p ) by t ⊆t, if the following conditions hold:
In this subsection, we discuss the relationship between the pr x d
(1) ∀t∈t−t ,d(t,p ,p )=d(t,p ,p ),
positional difference of two programs and their behavioral d r x r y
(2) ∀t∈t ,d(t,p ,p )=0,
difference.Let usconsidertwo arbitrarypositionsin thesame d r x
(3) ∀t∈t ,d(t,p ,p )=1,
program space, in terms of Definition 4. Because the position d r y
for all p ,p ∈P. This is written as dt (p )−t→d dt (p ) or
ofaprogramindicatesthebehavioraldifferenceoftheprogram x y pr x pr y
relative to the origin, it is expected that there is a relationship simply p −t→d p .
x y
between programs’ positions and behaviors. For example, if
In other words, (1) the position of p is the same as the
two programs are in different positions in one dimension, it x
positionofp inalldimensionsexceptfort ,(2)theposition
implies that the two programs have different behaviors for the y d
of p in t dimensions is the same as the origin, (3) the
corresponding test. This fact can be generalized as follows: x d
position of p in t dimensions is the opposite of the origin.
y d
(dtpr(px)(cid:54)=dtpr(py)) =⇒ (d(t,px,py)(cid:54)=0), (2) Here, td represents every test t ∈ t where d(t,px,py) = 1.
When the triple (t,p ,d) is not the main concern, we shorten
for all p ,p ,p ∈ P,d ∈ D, and t ∈ Tn, with arbitrary r
x y r the notation for the position of a program p to the position
n. In (2), the left-hand side (LHS) implies that p and p
x y (vector) p.
are in different positions in the program space of (t,p ,d).
r Thedeviancerelationindicateshowtestsinfluencepositions
The right-hand side (RHS) implies that p and p have dif-
x y inatestingprocess.Ingeneral,asetoftestsincreasesinsizeto
ferent behaviors for t. Roughly speaking, (2) implies that the
become more effective at detecting faults. This growth of the
difference of positions of programs guarantees the difference
testsetisdescribedbyt inDefinition5:t makesp deviant
d d y
of behaviors for the programs. The proof of this is omitted,
fromp .Forexample,ifacorrectprogramp isgivenbyp ,
x s x
becauseitistrivial.By(2),itissafetoconcludethatprograms
then t becomes the set of tests that detects faults in every
d
in different positions have different behaviors for a given set
program in p . On the other hands, if an original program
y
of tests in a space.
p is at p , then t becomes the set of tests that kills every
o x d
Note that the inverse of (2) does not hold. In other words,
mutant in p .
y
even if two programs are in the same position in a program
The deviance relation is asymmetric. For all positions
space,thisdoesnotimplythatthetwoprogramshavethesame
p ,p and tests t , the following is true:
x y d
behaviors for the tests corresponding to the program space.
This can be formalized as follows: p −t→d p =⇒ ¬(p −t→d p )
x y y x
(dtpr(px)=dtpr(py)) =(cid:54)⇒ (d(t,px,py)=0). (3) Interestingly, it is not only asymmetric but also transitive.
For all positions p ,p ,p and tests t ,t , the following is
Again, this holds for all p ,p ,p ∈ P,d ∈ D, and t ∈ Tn, x y z x y
x y r
true:
with arbitrary n. In (3), the reason why the LHS does not
imply the RHS is because of the case where px, py, and pr (px −t→x py)∧(py −t→y pz) =⇒ (px −t−x−o−r−t→y pz).
are all different to each other. An example of this is presented
in Figure 1. Consider t(cid:48) =(cid:104)t3(cid:105), where the three programs ps, In other words, all positions deviant from py are also deviant
po, and m have different behaviors. However, the position of from px if py is deviant from px. This transitivity of the
p relative to p is the same as the position of m relative to deviance relation on positions is closely related to the redun-
o s
p , because dt(cid:48)(p )=dt(cid:48)(m)=(cid:104)1(cid:105). dancy of mutants. For example, it may be the case that
s ps o ps
Tosummarize,thepositionofaprograminaprogramspace p −t→1 m −t→2 m −t→3 m →···−t→n m ,
0 1 2 3 n
indicatesitsbehavioraldifferencewithrespecttotheoriginof
the program space. The meaning of the position depends on foranoriginalprogramp ,mutantsm ,m ,··· ,m ,andtests
o 1 2 n
what program is used as the origin of the space. The meaning t ,t ,··· ,t . In this case, t is enough to make all positions
1 2 n 1
Fig.2. Positiondeviancegraphforathree-dimensionalprogramspace
Fig.3. GrowthofthePDLfromone-dimensiontothree-dimension
deviant from p because of the transitivity of the deviance
0
relation, and this leads that t1 is enough to kill all mutants. In PDL is useful for analyzing mutation-based methods. For
other words, m2,··· ,mn are redundant by m1 with respect example, we can analyze and improve the theoretical founda-
to the given tests in this case. Because there is a well-defined tion for the mutant set minimization recently established by
method to find redundant mutants, introduced by Ammann et Ammann et al. [27]. We will explain this process in Section
al.[27],weanalyzethisusingourframeworkinSectionIV-B. IV-B.
PDLisworthwhileforconsideringtherelationshipbetween
G. Position Deviance Lattice
the growth of tests and positions. To put it bluntly, PDL
Thedeviancerelationnaturallyformsalatticeofpositions3. grows by adding tests. Figure 3 demonstrates the growth of
We introduce the Position Deviance Lattice (PDL) which PDL from one-dimension to three-dimension. In Figure 3 (c),
showsthepositionsandtheirdeviancerelationshipsasfollows: the PDL is the same as Figure 2, which represents a three-
dimensional program space. Figure 3 (a) and (b) show the
growth of the PDL as tests are individually added. Note that
Definition 6: For a program space given by (t,p ,d), a
r the eight positions p ,··· ,p are expressed in each PDL.
position deviance lattice is a directed graph G = (N,E) 0 7
First, when the test set t has only one test t , p ,p ,p ,p
consists of 1 1 4 5 7
aredeviantfromp butp ,p ,p ,p arenotdevianttoeach
(1) N ={p | p=dt (p) for p∈PS}, 0 0 2 3 6
pr t other.Aftert2isadded,p2andp6aredeviantfromp0,butp3
(2) E ={(px,py) | px →− py for single t∈t} is not yet deviant from p0. Similarly, p4 and p7 are deviant
where (px,py) ∈ E refers to the directed edge from px to from p1,p5, and there are four different nodes in Figure 2
py. (b). Finally, t3 makes p3 deviant from p0, p5 from p1, p6
from p , and p from p , respectively.
In other words, (1) the set of nodes includes all possible 2 7 4
It is possible to consider testing process as the growth of
positions in the program space corresponding to (t,p ,d), (2)
r
PDL. For example, in mutation testing, the generation of new
thesetofedgesincludeseverydirectedpairofpositionshaving
testsandexecutionoflivemutantsarerepeateduntilthesetof
the deviance relation with a single test. PDL represents the
tests kills all of the mutants. This means that the generation
frame of the n-dimensional program space by its positions
of new dimensions is repeated until all of the positions of
and deviance relations. Note that an n-dimensional program
space contains 2n possible positions, because each dimension mutants are deviant from the origin.
Mathematically, PDL is a form of hypercube graph, with
contributes two positions.
2n nodes, 2n−1n edges, and n edges touching each node,
For example, Figure 2 presents the PDL that illustrates the
where n is the number of dimensions in the corresponding
frame of a three-dimensional program space containing eight
programspace.WhenPDLisappliedtomutation-basedtesting
positions. The arrows in the leftmost dashed-line box indicate
methods,severalmathematicalpropertiesofhypercubegraphs
the directions of the three dimensions t = (cid:104)t ,t ,t (cid:105). The
1 2 3
may guide for the elaboration of the mutation-based testing
main graph illustrates all of the possible positions p ,··· ,p
0 7
methods.
in the program space and their deviance relations. The values
in the rightmost side indicate positions’ distance from the
IV. APPLICATIONS
origin.
A. Mutation-based Fault Localization
3Strictly, the deviance relation provides a strict partial order which is not Program faults can be detected by fault detection methods
a lattice but a directed acyclic graph. However, it is easy to consider the such as mutation testing; once a fault is detected, its location
correspondingnon-strictpartialordergivenbythe“deviantfromorequalto”
must be analyzed so that the fault can be corrected. This
relation. This non-strict partial order is a lattice, and we simply saying that
thelatticeisformedbythedeviancerelationonpositions. can be very tedious and time consuming, especially in large
and complex programs. Among the many contributions to the m=m minus the likelihood of m=m .
f c
field of fault localization, Spectrum-Based Fault Localization In a program space, a test t that failed on p but passed on
(SBFL) has received significant attention, owing to its sim- m corresponds to a dimension t such that:
plicity and effectiveness [28]. A program spectrum contains
information recorded during the execution of a program, such d(cid:104)t(cid:105)(p)=1∧d(cid:104)t(cid:105)(m)=0
ps ps
as its executed statements. The spectrum is used to identify ⇔(d(cid:104)t(cid:105)(p)(cid:54)=d(cid:104)t(cid:105)(m))∧d(cid:104)t(cid:105)(m)=0.
suspicious statements that cause a program test to fail. The ps ps ps
basic idea is that executed statements that cause the failed Let t(cid:48) be a collection of t satisfying d(cid:104)t(cid:105)(p) (cid:54)= d(cid:104)t(cid:105)(m).
test are associated with the failure. For example, if only Then, n (m) is equal to the number opfszeros in dpts(cid:48)(m).
one executed statement is associated with a failed test, it is f→p ps
In the same manner, n (m) is equal to the number of
p→f
obvious that the executed statement caused the test to fail; ones in dt(cid:48)(m). For example, if dt(cid:48)(m) = (cid:104)0,1,0,0(cid:105), then
thus, the location of the fault is that statement. After many ps ps
n (m)=3 and n (m)=1.
f→p p→f
tests are executed, the suspiciousness value of each statement
Thissignifiesthatthesuspiciousnessvalueofmincreasesas
is calculated based on the similarity of the program spectrum
the position of m moves towards the origin p in the program
and the testing results (i.e., pass or fail). All statements are s
space. Note that the dimensions of the space are t(cid:48), not t.
orderedbytheirsuspiciousnessvalue.Thehigherastatement’s
In other words, Seokhyeon et al. [7] effectively found the m
suspiciousnessvalue,thehigheritsprobabilityofbeingfaulty.
close to p by focusing on the dimensions that caused the test
An ideal fault localization method ranks the faulty statement s
result changes (i.e., p→f or f →p) for p and m.
at the top, to allow programmers or even repair algorithms
2) Mutant as a fault: Papadakis and Traon [6] considered
to correct the fault. There are many suspiciousness (i.e.,
theconduct4 offaults(includingmutants)withregardtotests.
similarity) metrics, such as Tarantula [29], Ochiai [30], and
They affirmed that a mutant m has the same conduct as
Jaccard [31]. Recently, Xie et al. [28] developed a theoretical x
anothermutantm ifm andm arekilledbythesametests.
framework to analyze the efficacy of suspiciousness metrics. y x y
Thekeyassumptionof[6]isthatmutantsandfaultslocatedon
Recently, several researchers developed a new fault lo-
thesameprogramstatementwillshowsimilarmotions.Based
calization concept called Mutation-Based Fault Localization
onthisassumption,foramutantmandanunlocalizedfaultf,
(MBFL) [6], [7]. Similar to SBFL, MBFL calculates the
the location of f is given by m, whose conduct is similar to
suspiciousness value of each statement in a program, and
the actionof f. In other words, the suspiciousnessvalue of m
ranksthestatementsintheorderoftheirsuspiciousnessvalue.
is calculated according to the similarity between the conduct
The key feature of MBFL is that the suspiciousness of each
of m and the conduct of f.
statement is calculated according to the suspiciousness of the
Unfortunately,thisassumptionisinsufficientforcalculating
mutantsinthestatement.Becausemanymutantsaregenerated
the suspiciousness value of m when the conduct of faulty
from each statement in general, MBFL has finer granularity
program p is not clearly defined. In [6], the conduct of p is
than SBFL. o o
implicitly defined as test results (i.e., pass or fail) instead of
Interestingly, according to the experiments in [6], [7],
kill results. This signifies that the conduct of p is based on
MBFL has significant advantages over SBFL. However, an- o
p , while the conduct of m is based on p. The meaning of the
alyzing and discussing MBFL can be difficult, because of its s
similarity between the conduct of m and the conduct of p
lack of formal foundations. Thus, a program space is applied o
is ambiguous, because they have different bases. A program
to analyze and discuss the foundations of MBFL. Based on
space is applied to explore this ambiguousness.
the analysis results, there are two fundamental considerations
in MBFL: (1) considering a mutant as a potential fix, and (2) In a program space, the conduct of a mutant m is
considering a mutant as a fault.
dt (m)
1) Mutant as a partial fix: Seokhyeon et al. [7] considered po
twotypesofmutants:m ,whichrepresentsamutantgenerated
c
for an original program p , a set of tests (as a test vector) t,
by mutating a correct statement, and m , which represents o
f
and a differentiator d. This indicates that the conduct of m
a mutant generated by mutating a faulty statement. They
represents the position of the point of m relative to the origin
observed that failed tests on the original program p of the
p . On the other hand, based on the implicit definition, the
mutants are more likely to pass on m than on m . On the o
f c
conduct of a faulty program p is
otherhand,passedtestsonparemorelikelytofailonm than
c
on m . Let n (m) be the number of tests that failed on p
f f→p dt (p )
but passed on an arbitrary mutant m. Then, the proportion ps o
of n (m) over all failed tests implies the likelihood of
f→p for a correct program p , t, and d. This indicates that the
s
m = m . Similarly, let n (m) be the number of tests
f p→f conduct of p represents the position of p relative to the
o o
that passed on p but failed on an arbitrary mutant m; then,
the proportion of n (m) over all passed tests implies the
p→f 4In[6],thetermbehavior isused,notconduct.Becausebehavior isused
likelihood of m = m . Based on these observations, the
c as another means in this paper, here we use an alternative term conduct to
suspiciousness value of m is calculated by the likelihood of avoidconfusion.
TABLEI
EXAMPLE:MUTANTKILLINFORMATION
m1 m2 m3 m4
t1 1 0 1 1
t2 0 1 0 1
t3 0 1 1 1
(a) Mutantasapartialfix (b) Mutantasafault
Fig.4. TwofoundationsofMBFLrepresentedbyprogramspaces
origin p , not p . As a result, the suspiciousness value of m
s o
is calculated according to the following similarity:
dt (m)∼dt (p )
po ps o
=dt (m)∼dt (p ).
po po s
Fig.5. PDLwithmutantsforfindingaminimalsetofmutants
This signifies that the suspiciousness value of m represents
the proximity between the positions of m and the position of
p in the space whose origin is p . In other words, while it B. Mutant Set Minimization
s o
is not explained in [6], m and p are regarded as two faulty
s One long-standing problem that prevents mutation-based
programs based on p .
o testing from becoming practical is the high cost of executing
3) Implications: The conceptual foundations of MBFL are a very large number of mutants against a set of tests [32]. In
summarized in Figure 4. Each of the dotted circles represents terms of reducing cost, minimizing mutant sets by removing
a program space, and the crossed line at the center implies redundant mutants (with respect to given tests) is a promising
the origin of each space. Each program space contains the strategy. In this subsection, we introduce a mutant set mini-
three major points corresponding to po, m, and ps. The arrow mization method for a given set of tests, and apply a position
represents the method for calculating the suspiciousness value deviance lattice (PDL) to provide deeper implications.
ofm.Forsimplicity,MBFL-FIXreferstotheMBFLmethods Recently, Ammann et al. [27] established a theoretical
that consider a mutant as a partial fix, and MBFL-FLT refers foundation for mutant set minimization based on the formal
totheMBFLmethodsthatconsideramutantasafault.Figure relations of mutants, called dynamic subsumption. If a mutant
4(a) shows the foundation of MBFL-FIX, and Figure 4(a) m iskilledbyatleastonetestinasetoftestsTS andanother
x
shows the foundation of MBFL-FLT. mutant m is always killed whenever m is killed, then m
y x x
It is clearly shown that both MBFL methods focus on the dynamicallysubsumesm withrespecttoTS.Theyprovided
y
same objective: the similarity (or proximity) between m and that a mutant set M is minimal with respect to TS if and
min
ps. This indicates that MBFL is fundamentally the same as only if there does not exist a distinct pair mx,my ∈ Mmin
program repair, in which the objective is to move toward such that m dynamically subsumes m . This signifies that
x y
ps from p using m. This implication matches our intuition, if mx dynamically subsumes my, then my is redundant to
becausetheobjectiveoffaultlocalizationistofindthelocation m with respect to TS. For example, consider four mutants
x
that requires the correction. As m moves closer to ps, the m1,m2,m3,andm4, and a set of tests t = (cid:104)t1,t2,t3(cid:105). Table
location of m in the program is more likely to be the location I shows which mutants each test kills. Specifically, the (i,j)
requiring the correction. There are even cases in which m is elementofthetableisthevalueofd(t ,p ,m ):ift killsm
i o j i j
the direct correction of po in both MBFL methods [6], [7]. then 1, otherwise 0.
Regardingthedifferencesbetweenthetwomethods,MBFL- According to the definition of dynamic subsumption, m
1
FIX utilizes dt (m) ∼ 0, while MBFL-FLT utilizes dynamicallysubsumesm andm ,m dynamicallysubsumes
ps 3 4 2
dt (m) ∼ dt (p ). As analyzed in (3), dt (m) ∼ dt (p ) m , and m dynamically subsumes m . By removing all
po po s po po s 4 3 4
=(cid:54)⇒ dt (m)∼0. Specifically, if p ,p , and m are different dynamically subsumed mutants, M ={m ,m } becomes
ps o s min 1 2
fromeachotherinatestt,MBFL-FLTwilldeterminethatmis the minimal set of mutants with respect to the test set TS =
thesameasp fort,whileMBFL-FIXwilldeterminethatmis {t ,t ,t }. This provides a solid theoretical foundation for
s 1 2 3
differentfromp fort.ThisshowsthatMBFL-FIXis,atleast, mutantsetminimization.Weusedourtheoreticalframeworkto
s
better than MBFL-FLT for calculating the similarity between interpret this foundational study, and found more implications
m and p . We believe additional theoretical and experimental to elaborate the mutant set minimization.
s
studies will result in improved MBFL methods. Let us create a PDL based on Table I. As we previously
discussed in Section III-D, dt (m)=m implies which t∈t relationship between the size of M and TS. Yet, the
po min
kills m. From Table I, m = (cid:104)1,0,0(cid:105), m = (cid:104)0,1,1(cid:105), relationshipbetweenTS andM shouldbeinvestigatedfurther
1 2
m = (cid:104)1,0,1(cid:105), and m = (cid:104)1,1,1(cid:105). This provides the PDL, todetermine,forexample,themeaningoftheratioof|M |
3 4 min
as shown in Figure 5. The gray box refers to the position that to max(|M |), or the practical maximum of |M | with
min min
containsamutant.InthePDL,itiseasytoseethatm →m , respect to TS, based on the theoretical framework.
1 3
m → m , m → m , and m → m . Note that the
1 4 2 4 3 4
deviance relations between positions precisely correspond to V. CONCLUSION
thedynamicsubsumptionrelationsbetweenmutants.Formally,
In this paper, we considered a theoretical framework for
the following is true for all p ∈ P, m ,m ∈ M
o x y min better understanding of mutation-based testing methods. In
generated from p , t, and d:
o particular,wedefinedatestdifferentiatortoshifttheparadigm
of testing from the correctness of a program to the difference
p →m →m in a program space of (t,p ,d) (4)
o x y o
between programs. A test differentiator clearly and concisely
⇔∃td ⊆t (dtpo(mx)−t→d dtpo(my)) (5) represents the behavioral differences between programs in a
∧(dt (m )(cid:54)=0) (6) test. With regards to a test set, we defined a d-vector that
po x
represents the behavioral differences between two programs
⇔∀t∈t ((d(t,p ,m )=1) =⇒ (d(t,p ,m )=1)) (7)
o x o y
in vector form.
∧∃t∈t d(t,p ,m )=1 (8)
o x Using the fact that a vector can be regarded as representing
⇔mx dynamically subsumes my with respect to t (9) a point in a multidimensional space, we define the space of
programs corresponding to d-vectors. In the program space,
Briefly, (5) signifies m −t→d m for some t (cid:54)= ∅, and
x y d the position of a program relative to the origin in each
(6) indicates that m is killed by at least one test because
x dimension indicates the behavioral difference between the
its position is deviant from p . (7) corresponds to (5), and
o program and the origin for the test corresponding to the
(8) corresponds to (7). Consequently, (4) is equivalent to (9).
dimension. The relationship between different positions and
This signifies that the deviance relation on positions in a
behaviors is clearly addressed. We then continued to define
certainprogramspaceisthesameasthedynamicsubsumption
the derivation relation on positions for representing how tests
relation on mutants. In other words, deviance relations are a
influencepositionsinatestingprocess.Thepositionderivation
more general and comprehensive concept for representing the
lattice (PDL) is defined for providing visual aids for positions
behavioral differences of programs, while dynamic subsump-
and their derivation relations.
tion relations are a more specific concept for representing
We then revisited the existing mutation-based fault local-
the behavioral differences between mutants and an original
ization methods and the mutant set minimization method,
program. The comprehensiveness of deviance relations is also
demonstrating the applicability of our theoretical framework.
shown in Figure 5. If we remove all nodes that do not have
For mutation-based fault localization methods, we found that
a mutant from the PDL, the remaining graph will precisely
the common foundation is the proximity between mutants
represent the dynamic mutant subsumption graph (DMSG)
and the correct program in the program space, while the
introduced by Kurtz et al. [33]. The DMSG makes it easy
method of calculating the proximity is different. We also
to identify M , which is simply the set of root nodes; of
min foundthatonemethodis,atleast,theoreticallybetterthanthe
course, the PDL provides the same benefit.
other method. Furthermore, we showed that our theoretical
The PDL provides more implications than the DMSG. In
framework is sufficiently general to include all theoretical
Figure 5, the maximum number of positions that are not
foundationsformutantsetminimization.Wealsoimprovedthe
deviantfromtheotheristhree.Inotherwords,foranarbitrary
mutant set minimization theory by providing the theoretical
four (or more) positions, at least one position is deviant from
maximum size of a minimal mutant set. Given our results,
another position. This indicates that the theoretical maximum
we demonstrated that our theoretical framework may serve
size of M is three, with respect to any test set |TS| = 3.
min as a solid foundation for discussions in both empirical and
In general, the maximum size of M is
min theoretical studies on mutation-based testing.
(cid:18) (cid:19)
n
max(|M |)=
min (cid:98)n/2(cid:99) REFERENCES
where n is the size of a test set. In other words, for an [1] R. A. DeMillo, R. J. Lipton, and F. G. Sayward, “Hints on test data
selection:Helpforthepracticingprogrammer,”Computer,vol.11,no.4,
arbitrary test set TS and an arbitrary mutants set M, the
pp.34–41,1978.
theoretical maximum size of the minimal mutant set Mmin [2] J. H. Andrews, L. C. Briand, Y. Labiche, and A. S. Namin, “Using
with respect to TS is (cid:0) n (cid:1). Further, max(|M |) is given mutationanalysisforassessingandcomparingtestingcoveragecriteria,”
(cid:98)n/2(cid:99) min Software Engineering, IEEE Transactions on, vol. 32, no. 8, pp. 608–
without executing all m ∈ M against TS. For example, for
624,2006.
|TS|=5, max(|Mmin|) is (cid:0)52(cid:1)=10. This signifies that, with [3] N.Li,U.Praphamontripong,andJ.Offutt,“Anexperimentalcomparison
respect to the given TS, any mutant set M will be reduced of four unit test criteria: Mutation, edge-pair, all-uses and prime path
coverage,” in Software Testing, Verification and Validation Workshops,
to contain at most 10 mutants. This improves the theoretical
2009.ICSTW’09.InternationalConferenceon. IEEE,2009,pp.220–
foundation for mutant set minimization by highlighting the 229.
[4] R.Just,D.Jalali,L.Inozemtseva,M.D.Ernst,R.Holmes,andG.Fraser, (ICST),2014IEEESeventhInternationalConferenceon. IEEE,2014,
“Are mutants a valid substitute for real faults in software testing?” in pp.21–30.
Proceedings of the 22nd ACM SIGSOFT International Symposium on [28] X. Xie, T. Y. Chen, F.-C. Kuo, and B. Xu, “A theoretical analysis
FoundationsofSoftwareEngineering. ACM,2014,pp.654–665. of the risk evaluation formulas for spectrum-based fault localization,”
[5] E. Schulte, Z. P. Fry, E. Fast, W. Weimer, and S. Forrest, “Software ACMTransactionsonSoftwareEngineeringandMethodology(TOSEM),
mutationalrobustness,”GeneticProgrammingandEvolvableMachines, vol.22,no.4,p.31,2013.
vol.15,no.3,pp.281–312,2014. [29] J. A. Jones and M. J. Harrold, “Empirical evaluation of the tarantula
[6] M. Papadakis and Y. Le Traon, “Metallaxis-fl: mutation-based fault automatic fault-localization technique,” in Proceedings of the 20th
localization,”SoftwareTesting,VerificationandReliability,2013. IEEE/ACM international Conference on Automated software engineer-
[7] S. Moon, Y. Kim, M. Kim, and S. Yoo, “Ask the mutants: Mutating ing. ACM,2005,pp.273–282.
faultyprogramsforfaultlocalization,”inSoftwareTesting,Verification [30] R. Abreu, P. Zoeteweij, and A. J. Van Gemund, “An evaluation of
andValidation(ICST),2014IEEESeventhInternationalConferenceon. similarity coefficients for software fault localization,” in Dependable
IEEE,2014,pp.153–162. Computing,2006.PRDC’06.12thPacificRimInternationalSymposium
[8] V.DebroyandW.E.Wong,“Usingmutationtoautomaticallysuggest on. IEEE,2006,pp.39–46.
fixesforfaultyprograms,”inSoftwareTesting,VerificationandValida- [31] M.Y.Chen,E.Kiciman,E.Fratkin,A.Fox,andE.Brewer,“Pinpoint:
tion(ICST),2010ThirdInternationalConferenceon. IEEE,2010,pp. Problemdeterminationinlarge,dynamicinternetservices,”inDepend-
65–74. ableSystemsandNetworks,2002.DSN2002.Proceedings.International
[9] C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer, “Genprog: A Conferenceon. IEEE,2002,pp.595–604.
generic method for automatic software repair,” Software Engineering, [32] Y.JiaandM.Harman,“Ananalysisandsurveyofthedevelopmentof
IEEETransactionson,vol.38,no.1,pp.54–72,2012. mutationtesting,”SoftwareEngineering,IEEETransactionson,vol.37,
[10] V.DebroyandW.E.Wong,“Combiningmutationandfaultlocalization no.5,pp.649–678,2011.
for automated program debugging,” Journal of Systems and Software, [33] B.Kurtz,P.Ammann,M.E.Delamaro,J.Offutt,andL.Deng,“Mutant
vol.90,pp.45–60,2014. subsumption graphs,” in Software Testing, Verification and Validation
[11] J. S. Gourlay, “A mathematical framework for the investigation of Workshops (ICSTW), 2014 IEEE Seventh International Conference on.
testing,” Software Engineering, IEEE Transactions on, no. 6, pp. 686– IEEE,2014,pp.176–185.
709,1983.
[12] J. B. Goodenough and S. L. Gerhart, “Toward a theory of test data
selection,”SoftwareEngineering,IEEETransactionson,no.2,pp.156–
173,1975.
[13] W. E. Howden, “Reliability of the path analysis testing strategy,”
SoftwareEngineering,IEEETransactionson,no.3,pp.208–215,1976.
[14] E.J.WeyukerandT.J.Ostrand,“Theoriesofprogramtestingandthe
the application of revealing subdomains.” Software Engineering, IEEE
Transactionson,vol.6,no.3,pp.236–246,1980.
[15] M.Staats,M.W.Whalen,andM.P.E.Heimdahl,“Programs,tests,and
oracles: the foundations of testing revisited,” in Software Engineering
(ICSE),201133rdInternationalConferenceon. IEEE,2011,pp.391–
400.
[16] M.Staats,S.Hong,M.Kim,andG.Rothermel,“Understandinguserun-
derstanding: determining correctness of generated program invariants,”
inProceedingsofthe2012InternationalSymposiumonSoftwareTesting
andAnalysis. ACM,2012,pp.188–198.
[17] N.Polikarpova,C.A.Furia,Y.Pei,Y.Wei,andB.Meyer,“Whatgood
are strong specifications?” in Proceedings of the 2013 International
ConferenceonSoftwareEngineering. IEEEPress,2013,pp.262–271.
[18] G.FraserandA.Arcuri,“Achievingscalablemutation-basedgeneration
of whole test suites,” Empirical Software Engineering, vol. 20, no. 3,
pp.783–812,2014.
[19] E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The
oracleprobleminsoftwaretesting:Asurvey,”2015.
[20] A. J. Offutt, “Investigations of the software testing coupling effect,”
ACMTransactionsonSoftwareEngineeringandMethodology(TOSEM),
vol.1,no.1,pp.5–20,1992.
[21] T.A.Budd,R.A.DeMillo,R.J.Lipton,andF.G.Sayward,“Theoretical
andempiricalstudiesonusingprogrammutationtotestthefunctional
correctness of programs,” in Proceedings of the 7th ACM SIGPLAN-
SIGACTsymposiumonPrinciplesofprogramminglanguages. ACM,
1980,pp.220–233.
[22] T. A. Budd and D. Angluin, “Two notions of correctness and their
relationtotesting,”ActaInformatica,vol.18,no.1,pp.31–45,1982.
[23] L. J. Morell, “A theory of fault-based testing,” Software Engineering,
IEEETransactionson,vol.16,no.8,pp.844–857,1990.
[24] M.Harman,Y.Jia,andW.B.Langdon,“Stronghigherordermutation-
basedtestdatageneration,”inProceedingsofthe19thACMSIGSOFT
symposium and the 13th European conference on Foundations of soft-
wareengineering. ACM,2011,pp.212–222.
[25] J.Offutt,“Amutationcarol:Past,presentandfuture,”Informationand
SoftwareTechnology,vol.53,no.10,pp.1098–1107,2011.
[26] M. Woodward and K. Halewood, “From weak to strong, dead or
alive?ananalysisofsomemutationtestingissues,”inSoftwareTesting,
Verification,andAnalysis,1988.,ProceedingsoftheSecondWorkshop
on. IEEE,1988,pp.152–158.
[27] P. Ammann, M. E. Delamaro, and J. Offutt, “Establishing theoretical
minimalsetsofmutants,”inSoftwareTesting,VerificationandValidation