umi-umd-3801

165 Pages·2012·1.93 MB·English

ABSTRACT
Titleofdissertation: Developing Cost-Effective Model-Based Techniques
for GUI Testing
Qing Xie, Doctor of Philosophy, 2006
Dissertationdirectedby: Professor Atif Memon
Department of Computer Science
University of Maryland, College Park
Most of today’s software users interact with the software through a graphical user
interface (GUI), which constitutes as much as 45-60% of the total code. The correct-
ness of the GUI is necessary to ensure the correctness of the overall software. Although
GUIshavebecomeubiquitous,testingGUIsforfunctionalcorrectnesshasremainedane-
glected research area. Existing GUI testing techniques are extremely resource intensive
primarily because GUIs have very large input spaces and evolve frequently. This dis-
sertation overcomes the limitations of existing techniques by developing a process with
supporting models, techniques, and tools for continuous integration testing of evolving
GUI-based applications. The keyidea of thisprocess is tocreate three concentric testing
loops, each with speci(cid:2)c GUI testing goals, resource usage, and targeted feedback. The
innermost fully automatic loop called crash testing operates on each code change of the
GUI software. The second semi-automated loop called smoke testing operates on each
day’sGUI build. The outermostloop calledcomprehensive GUI testingis executedafter
amajorversionoftheGUI isavailable. Theprimaryenablersofthisprocess,alsodevel-
oped in this dissertation, include an abstract model of the GUI and a set of model-based
techniquesfortest-casegeneration,testoraclecreation,andcontinuousGUItesting. The
model and techniques were obtained by studying GUI faults, interactions between GUI
events, and why certain event interactions lead to faults. The continuous testing process
andassociatedtechniquesareshowntobeuseful,viaseverallargeexperimentsinvolving
millionsoftestcases,onbothin-houseandopen-sourceGUIapplications.
Developing Cost-Effective Model-Based Techniques for
GUI Testing
by
Qing Xie
DissertationsubmittedtotheFacultyoftheGraduateSchoolofthe
UniversityofMaryland,CollegeParkinpartialful(cid:2)llment
oftherequirementsforthedegreeof
DoctorofPhilosophy
2006
AdvisoryCommittee:
ProfessorAtifMemon,Chair/Advisor
ProfessorVictorBasili
ProfessorRance Cleaveland
ProfessorMichelCukier
ProfessorAdamPorter
ProfessorGangQu
(cid:0)
c Copyright by
Qing Xie
2006
ACKNOWLEDGMENTS
First and foremost I would like to thank my advisor, Professor Atif Memon for
giving me an invaluable opportunity to work on challenging and extremely interesting
projects over the past several years. He has always made himself available for help and
advice and there has never been an occasion when I have knocked on his door and he
has not given me time. It has been a pleasure to work with and learn from such an
extraordinaryindividual. Thisexperiencewillbene(cid:2)tmeforever.
Thanks are due to Professor Victor Basili, Professor Rance Cleaveland, Professor
Michel Cukier, Professor Adam Porter, and Professor Gang Qu for agreeing to serve on
mythesiscommitteeandforsparingtheirinvaluabletimereviewingthemanuscript.
I thank all my teachers in schools, colleges, and universitieswhose dedication and
hardworkhelpedlaythefoundationforthiswork.
My colleagues at the Software Testing Group have enriched my graduate life in
many ways and deserve a special mention. My interaction with Bin Gan, Xun Yuan,
JaymieStrecker,CyntricaEaton,AdithyaNagarajan,IshanBanerjee,andLorinHochstein
hasbeenveryfruitful.
Iwouldalsoliketoacknowledgehelpandsupportfromsomeofthestaffmembers
inthedepartment.
I owe my deepest thanks to my family - my mother, father, and brother who have
always stood by me and guided me through my career; my husband, Feng, for always
ii
beingtheretosupportme andbeconstantsourceofencouragementduringmyPh.D;my
daughter,Katie,forbringingmeendlesshappiness.
I would like to acknowledge (cid:2)nancial support from the NSF grant CCF-0447864
and the Of(cid:2)ce of Naval Research grant N00014-05-1-0421,for all the projectsdiscussed
herein.
Last, but far from least, I want to express my thanks to all the people who have
madethisthesispossibleandbecauseofwhommygraduateexperiencehasbeenonethat
Iwillcherishforever.
iii
TABLEOFCONTENTS
ListofTables vi
ListofFigures vii
1 Introduction 1
1.1 WhatisaGUI? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 GUITestingProcess . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 ChallengesofGUITesting . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 ExistingApproachesandtheirLimitations . . . . . . . . . . . . . . . . . 7
1.5 ANewContinuousGUITestingProcess . . . . . . . . . . . . . . . . . . 9
1.6 StructureoftheDissertation . . . . . . . . . . . . . . . . . . . . . . . . 12
2 BackgroundandRelatedWork 13
2.1 GUIRepresentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.1.1 GUI’sState . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.1.2 Event-(cid:3)owGraphs . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2 TestCase Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2.1 ManualApproaches . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2.2 Model-basedApproaches. . . . . . . . . . . . . . . . . . . . . . 22
2.3 TestOracles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.4 TestCoverageCriteria . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.5 Regressiontesting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.6 RapidFeedback-based QAmechanisms . . . . . . . . . . . . . . . . . . 29
2.7 FaultSeeding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3 AContinuousGUITestingProcess 33
3.1 InnermostLoop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.2 IntermediateLoop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.3 OutermostLoop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.4 InstantiatingtheLoops . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4 Crash Testing 40
4.1 MinimizedEffectiveEventContext . . . . . . . . . . . . . . . . . . . . 41
4.2 PilotStudy-UnderstandingtheMEEC . . . . . . . . . . . . . . . . . . . 42
4.2.1 StudyProcedure . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.2.2 Step1: StudySubjects . . . . . . . . . . . . . . . . . . . . . . . 44
4.2.3 Step2: FaultSeeding . . . . . . . . . . . . . . . . . . . . . . . . 45
4.2.4 Step3: Test-CaseGeneration . . . . . . . . . . . . . . . . . . . . 48
4.2.5 Step5: StudyingPredecessorEvents . . . . . . . . . . . . . . . . 51
4.3 DissectingtheMEEC . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.4 ThreatstoValidity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
iv
4.5 Event-InteractionGraph . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.6 CrashTestCases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.7 FeasibilityStudies-EvaluatingCrashTestCases . . . . . . . . . . . . . 65
4.7.1 FeasibilityStudy- CrashTestingonTerpOf(cid:2)ce Applications . . . 65
4.7.2 FeasibilityStudy- CrashTestingforOpen-SourceApplications . 75
4.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
5 SmokeTesting 86
5.1 DesigningDifferentTestOracles . . . . . . . . . . . . . . . . . . . . . . 87
5.1.1 Oracle Information . . . . . . . . . . . . . . . . . . . . . . . . . 87
5.1.2 Oracle Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.2 EvaluatingtheGUITestOracles . . . . . . . . . . . . . . . . . . . . . . 93
5.2.1 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . 93
5.2.2 ModelingCostandFaultDetectionEffectiveness . . . . . . . . . 94
5.2.3 ExperimentationProcedure . . . . . . . . . . . . . . . . . . . . . 96
5.2.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
5.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6 ComprehensiveGUITesting 115
6.1 Experiment - Studying the Characteristics of a (cid:147)Good(cid:148) Comprehensive
TestSuite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
6.1.1 ExperimentationProcedure . . . . . . . . . . . . . . . . . . . . . 116
6.1.2 TestPool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
6.1.3 Part1: EffectofTestSuiteSize . . . . . . . . . . . . . . . . . . 120
6.1.4 Part2: EffectofTestCaseLength . . . . . . . . . . . . . . . . . 123
6.1.5 Part3: EffectofEventComposition . . . . . . . . . . . . . . . . 126
6.2 Experiment-DevelopingTestOraclesforComprehensiveTesting . . . . 130
6.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
7 SummaryandFutureWork 138
7.1 SummaryofContributions . . . . . . . . . . . . . . . . . . . . . . . . . 138
7.2 FutureWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
Bibliography 146
v
LISTOFTABLES
4.1 TerpOf(cid:2)ceApplications . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.2 ClassesofSeeded Faults . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.3 SeededFaultsClassi(cid:2)edbyFunctionality . . . . . . . . . . . . . . . . . 47
4.4 RegularExpressionTable . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.5 SizesofEvent-InteractionsGraph . . . . . . . . . . . . . . . . . . . . . 67
4.6 NumberofTestCases GeneratedforEachVersionofEachApplication . 77
4.7 NumberofCrashesDetectedforEachVersionofEachApplication . . . . 77
5.1 TheDataTableCleanupSteps . . . . . . . . . . . . . . . . . . . . . . . 100
5.2 FriedmanTestResults . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
5.3 WilcoxonTestResults . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
5.4 AverageNumberofWidgetComparisonsPer TestCase . . . . . . . . . . 110
vi
LISTOFFIGURES
1.1 DifferentLoopsofContinuousGUITesting . . . . . . . . . . . . . . . . 11
2.1 (a)OpenGUI,(b)itsPartialState . . . . . . . . . . . . . . . . . . . . . 16
2.2 ExampleofanEvent-FlowGraph . . . . . . . . . . . . . . . . . . . . . 18
2.3 (a)ASimpleGUIand(b)ExampleofaJFCUnitTestCase . . . . . . . . 20
2.4 AnOverviewoftheGUIOracle . . . . . . . . . . . . . . . . . . . . . . 26
3.1 ActivitiestoSupportContinuousGUITesting . . . . . . . . . . . . . . . 38
4.1 TotalNumberofEventSequences . . . . . . . . . . . . . . . . . . . . . 48
4.2 EventDistribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.3 EventsInteractions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.4 MEECforTerpCalc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.5 MEECforTerpPaint . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.6 MEECforTerpSpreadSheet . . . . . . . . . . . . . . . . . . . . . . . . 54
4.7 MEECforTerpWord . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.8 EIGfor theEFGofFigure2.2 . . . . . . . . . . . . . . . . . . . . . . . 62
4.9 GenerateEvent-InteactionGraphfromEvent-FlowGraph . . . . . . . . . 63
4.10 TotalExecutionTime . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.11 NumberofSoftwareCrashes . . . . . . . . . . . . . . . . . . . . . . . . 68
4.12 NumberofCrash-CausingBugs . . . . . . . . . . . . . . . . . . . . . . 69
4.13 NumberofBugsvs. NumberofTestCases . . . . . . . . . . . . . . . . 70
4.14 EffectivenessoftheRotatingAlgorithmforTerpCalc . . . . . . . . . . . 72
4.15 EffectivenessoftheRotatingAlgorithmforTerpPaint . . . . . . . . . . . 73
4.16 EffectivenessoftheRotatingAlgorithmforTerpPresent . . . . . . . . . . 74
vii