Test Advising Framework

113 Pages·2013·0.74 MB·English

UUnniivveerrssiittyy ooff NNeebbrraasskkaa -- LLiinnccoollnn
DDiiggiittaallCCoommmmoonnss@@UUnniivveerrssiittyy ooff NNeebbrraasskkaa -- LLiinnccoollnn
Computer Science and Engineering: Theses, Computer Science and Engineering, Department
Dissertations, and Student Research of
Spring 2-4-2013
TTeesstt AAddvviissiinngg FFrraammeewwoorrkk
Yurong Wang
University of Nebraska-Lincoln, [email protected]
Follow this and additional works at: https://digitalcommons.unl.edu/computerscidiss
Part of the Computer Engineering Commons
Wang, Yurong, "Test Advising Framework" (2013). Computer Science and Engineering: Theses,
Dissertations, and Student Research. 54.
https://digitalcommons.unl.edu/computerscidiss/54
This Article is brought to you for free and open access by the Computer Science and Engineering, Department of at
DigitalCommons@University of Nebraska - Lincoln. It has been accepted for inclusion in Computer Science and
Engineering: Theses, Dissertations, and Student Research by an authorized administrator of
DigitalCommons@University of Nebraska - Lincoln.
TESTADVISINGFRAMEWORK
by
YurongWang
ATHESIS
PresentedtotheFacultyof
TheGraduateCollegeattheUniversityofNebraska
InPartialFulfilmentofRequirements
FortheDegreeofMasterofScience
Major: ComputerScience
UndertheSupervisionofProfessorsMatthewDwyerandSebastianElbaum
Lincoln,Nebraska
January,2013
TESTADVISINGFRAMEWORK
YurongWang,M.S.
UniversityofNebraska,2013
Advisors: MatthewDwyerandSebastianElbaum
Test cases are represented in various formats depending on the process, the technique or
the tool used to generate the tests. While different test case representations are necessary,
this diversity challenges us in comparing test cases and leveraging strengths among them -
acommontestrepresentationwillhelp.
Inthisthesis,wedefineanewTestCaseLanguage(TCL)thatcanbeusedtorepresent
test cases that vary in structure and are generated by multiple test generation frameworks.
Wealsopresentamethodologyfortransformingtestcasesofvaryingrepresentationsintoa
commonformatwheretheycanbematchedandanalyzed. Withthecommonrepresentation
inourtestcasedescriptionlanguage,wedefinefiveadvicefunctionstoleveragethetesting
strength fromone typeof tests toimprove theeffectiveness ofother type(s) oftests. These
advice functions analyze test input values, method call sequences, or test oracles of one
source test suite to derive advice, and utilize the advice to amplify the effectiveness of
an original test suite. Our assessment shows that the amplified test suite derived from
the advice functions has improved values in terms of code coverage and mutant kill score
comparedtotheoriginaltestsuitebeforetheadvicefunctionsapplied.
iii
ACKNOWLEDGMENTS
First of all, I would like to thank my advisers, Dr. Matthew Dwyer and Dr. Sebastian
Elbaum. Thisworkcouldnothavebeenaccomplishedwithouttheirsupportandguidance.
Dr. DwyerisverysincereandencouragingandDr. Elbaumissopatientandunderstanding.
I want to thank them for the great effort they put in to guide me through my research path
fromaverybasiclevel. Iamdeeplygratefulforthegreatopportunitytoworkboththem.
I also want to thank Dr. Suzette Person for all she’s done to help me with this work.
She is so nice, thoughtful and always ready to help when I have a problem. I would also
like to thank my committee member, Dr. Myra Cohen for offering great suggestions and
takingtimetoreadthisthesis.
Friendsin ESQuaReDhelped somuch duringmy courseof study. I thankthem forthe
fruitful discussion on research ideas, the help to tackle a technical issue, and all the small
talks among girls. I also want to thank Shelley Everett, Deb Heckens, Charles Daniel and
SheaSvobodafortheirgeneroushelpontheadministrativetasks.
Finally, I want to thank my family for their continuous support throughout my course
ofstudy.
iv
Contents
Contents iv
ListofFigures viii
ListofTables x
1 Introduction 1
1.1 MotivatingExample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 ResearchContributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3 ThesisOverview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2 RelatedWork 9
2.1 TheoreticalModelsofTesting . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 TestgenerationApproachesandTools . . . . . . . . . . . . . . . . . . . . 11
2.3 TestCaseRepresentations . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.4 TestAdvice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.5 TestevaluationTools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3 AFrameworkforRepresentingTests 20
3.1 ACommonTestRepresentation . . . . . . . . . . . . . . . . . . . . . . . 22
3.1.1 TCLSyntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
v
3.2 TransformingTestCasesTotheFramework . . . . . . . . . . . . . . . . . 27
3.2.1 ManualTeststoTCL . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2.2 JUnitTestsinTCL . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.2.3 DifferentialUnitTestinTCL . . . . . . . . . . . . . . . . . . . . . 32
3.2.4 AbstractJPFTestsinTCL . . . . . . . . . . . . . . . . . . . . . . 34
3.2.5 LoadTestsinTCL . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.4 AssessingtheTCL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4 TestAdvice 45
4.1 TestAdviceFunctions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.2 AFamilyofAdviceFunctions . . . . . . . . . . . . . . . . . . . . . . . . 48
4.2.1 AdviceonTestInputValues . . . . . . . . . . . . . . . . . . . . . 49
4.2.1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.2.1.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.2.1.3 FromHand-CodedTeststoRandomlyGeneratedTests . . 52
4.2.1.4 From Systematically Generated Tests to Randomly Gen-
eratedTests . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.2.2 TestMethodSequenceAdvice . . . . . . . . . . . . . . . . . . . . 56
4.2.2.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.2.2.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.2.2.3 CombiningSystem-levelRandomConcreteExecutionSe-
quenceswithUnit-levelSymbolicExecutionSequences . 57
4.2.3 AdviceonTestOracles . . . . . . . . . . . . . . . . . . . . . . . . 61
4.2.3.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.2.3.2 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . 61
vi
4.2.3.3 OracleAdvicefromHand-CodedTeststoOthers . . . . . 62
5 Evaluation 66
5.1 ResearchQuestions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5.2 Artifacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.3 InputAdvice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.3.1 FromHand-CodedTeststoRandomlyGeneratedTests . . . . . . . 67
5.3.1.1 ExperimentalDesign . . . . . . . . . . . . . . . . . . . . 68
5.3.1.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5.3.2 FromSystematicallyGeneratedTeststoRandomlyGeneratedTests 73
5.3.2.1 ExperimentDesign . . . . . . . . . . . . . . . . . . . . . 73
5.3.2.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
5.3.2.3 SyntheticProgramSnippet . . . . . . . . . . . . . . . . . 76
5.3.3 DiscussiononInputAdvice . . . . . . . . . . . . . . . . . . . . . . 77
5.4 SequenceAdvice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
5.4.1 Combining System-level Random Concrete Execution Sequences
withUnit-levelSymbolicExecutionSequences . . . . . . . . . . . 78
5.4.1.1 ExperimentalDesign . . . . . . . . . . . . . . . . . . . . 79
5.4.1.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
5.4.1.3 Discussion on Combining Random and Systematic Ap-
proaches . . . . . . . . . . . . . . . . . . . . . . . . . . 83
5.5 OracleAdvice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
5.5.1 OracleAdvicefromHand-CodedTeststoOthers . . . . . . . . . . 84
5.5.1.1 ExperimentalProcedure . . . . . . . . . . . . . . . . . . 85
5.5.1.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
5.6 ThreatstoValidity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
vii
6 ConclusionsandFutureWork 89
Bibliography 91
viii
List of Figures
1.1 SUT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Developer’sTestinScriptingLanguage . . . . . . . . . . . . . . . . . . . . . 3
1.3 TestgeneratedwithRandoop . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Test“specs”generatedwithSPF . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.1 TCLsyntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.2 TestexampleforInvokeAction . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.3 TestexampleforWriteAction . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.4 SodaVendingMachineTestCase . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.5 ExampletoShowHowTCLRepresentstheJUnitTests . . . . . . . . . . . . . 30
3.6 ExampletoShowHowTCLRepresentstheDUTs(pre-state) . . . . . . . . . . 32
3.7 ExampletoShowHowTCLRepresentstheDUTs(post-state) . . . . . . . . . 33
3.8 ExampletoShowHowTCLRepresentsAbstractTestCases-sourcecode . . . 35
3.9 ExampletoShowHowTCLRepresentsAbstractTestCases-pathconstraint . 35
3.10 SodaVendingMachineLoadTest . . . . . . . . . . . . . . . . . . . . . . . . 36
3.11 RandoopTestwithtry/catchBlock . . . . . . . . . . . . . . . . . . . . . . . . 39
3.12 Hand-codeteststranslatedfromJMLwithifbranch . . . . . . . . . . . . . . . 40
4.1 TestAdvisingApproach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.2 TestAdvicesbasedonTestAdvisingFramework . . . . . . . . . . . . . . . . 49
ix
4.3 Exampletodemonstratetheimportanceofinputvalueselection . . . . . . . . 50
4.4 Process to Generate Randoop Tests with and without input advice from Hand-
codedTests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.5 Relationshipbetweent inT andT inT . . . . . . . . . . . 58
randoop randoop amp amp
4.6 ProcesstoGenerateAmplifiedTestSuiteusingMethodSequenceAdvice . . . 60
4.7 ImplementationProcessoftheOracleAdvicefromhand-codedteststoothers . 64
4.8 ExampleToShowOracleAdviceGeneratedByOracleAnalyzer . . . . . . . . 64
5.1 ExampleofSpecificStringNeededinRandoop’sTestGeneration . . . . . . . 70
5.2 Exampletodemonstratetheimportanceofinputvalues . . . . . . . . . . . . . 75
5.3 Exampletodemonstratetheimportanceofinputvalues . . . . . . . . . . . . . 77
5.4 ExampletargetmethodwithNullPointerException . . . . . . . . . . . . . . . 83
5.5 ExampletoshowtherandomnessinmutantscorethatexistsinT . . . . 87
randoop