Dynamic State Alteration Techniques for Automatically Locating

247 Pages·2012·1.06 MB·English

UNIVERSITY OF CALIFORNIA
RIVERSIDE
Dynamic State Alteration Techniques for Automatically Locating Software Errors
A Dissertation submitted in partial satisfaction
of the requirements for the degree of
Doctor of Philosophy
in
Computer Science
by
Dennis Bernard Je(cid:11)rey
August 2009
Dissertation Committee:
Dr. Rajiv Gupta, Chairperson
Dr. Gianfranco Ciardo
Dr. Iulian Neamtiu
Copyright by
Dennis Bernard Je(cid:11)rey
2009
The Dissertation of Dennis Bernard Je(cid:11)rey is approved:
Committee Chairperson
University of California, Riverside
Acknowledgments
I would like to sincerely thank my advisors, Dr. Neelam Gupta and Dr. Rajiv Gupta,
for supporting me throughout this long journey. Six years ago, I knew nothing about
graduate studies or research. My advisors were steadfast in providing me with invaluable
help, guidance, and inspiration, and I never would have made it this far without them.
They believed in me during times when I did not believe in myself. Most importantly, they
deeply care about their students. I am immensely fortunate to be able to call them my
advisors.
I would like to thank my other dissertation committee members, Dr. Gianfranco
Ciardo and Dr. Iulian Neamtiu, for taking time out of their schedules to help me through
this last stage of my journey. I know this dissertation will be all the better because of them.
I would also like to thank my lab-mates, particularly Vijay Nagarajan, Chen Tian,
and Min Feng, for helping me in many ways during the last several years. They have been
great friends and together we have shared many memories that I will never forget.
I would like to extend a general thank you to all of the other teachers I have had
throughout my life. I am where I am today because of them.
Finally, I would like to thank my family. They are the ones who support me
unconditionally. Always have, always will.
iv
This dissertation is dedicated to my family:
past, present, and future.
v
ABSTRACT OF THE DISSERTATION
Dynamic State Alteration Techniques for Automatically Locating Software Errors
by
Dennis Bernard Je(cid:11)rey
Doctor of Philosophy, Graduate Program in Computer Science
University of California, Riverside, August 2009
Dr. Rajiv Gupta, Chairperson
Software does not always behave as expected due to errors. These errors can potentially
lead to disastrous consequences. Unfortunately, debugging software errors can be di(cid:14)cult
and time-consuming. Many techniques to automatically locate errors have been developed,
but the results are far from ideal. Unlike other techniques that analyze existing state
information from program executions, dynamic state alteration techniques modify the state
ofprogramexecutionstogaindeeperinsightintothepotentiallocationsoferrors. However,
prior state alteration techniques are generally no more e(cid:11)ective than other techniques, and
come at the expense of increased computation time. This dissertation shows that aggressive
and well-targeted state alteration techniques can be both highly e(cid:11)ective and reasonably
e(cid:14)cient.
The Value Replacement technique performs aggressive state alterations to locate
software errors by replacing the set of values used in di(cid:11)erent statement instances in failing
program executions. In a set of benchmarks, Value Replacement precisely identi(cid:12)es a faulty
statementin39outof129cases,whereasthemoste(cid:11)ectivetechniquepreviouslyknowndoes
so in 5 cases. Value Replacement can be generalized to iteratively locate multiple errors. A
vi
brute-force implementation of Value Replacement can require hours to locate a single error,
but techniques are developed that can reduce this timing requirement to minutes to locate
multiple errors.
The Execution Suppression technique performs targeted state alterations to lo-
cate memory errors by iteratively suppressing (avoiding) the e(cid:11)ects of statements involving
known memory corruption during failing executions. The technique is able to precisely
identify the (cid:12)rst point of memory corruption in all analyzed benchmark programs; this
point is typically at or close to the location of a memory error. Execution Suppression can
be generalized to locate multithreading errors including data races. While a software-only
implementation of suppression incurs an overhead of 7.2x on average, this overhead can be
reduced to 1.8x using hardware support.
Finally, a machine learning technique called BugFix is developed that provides
automatedassistanceinmodifyingafaultystatementto(cid:12)xanerror. Acasestudyillustrates
the potential bene(cid:12)t of the technique.
vii
Table of Contents
List of Figures xi
List of Tables xiii
1 Introduction 1
1.1 The Problem of Software Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Handling Software Errors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.1 Preventing Errors versus Responding to Errors . . . . . . . . . . . . . . . . . . 3
1.2.2 Overview of Software Testing and Debugging . . . . . . . . . . . . . . . . . . . 5
1.2.3 Approaches for Automating Software Debugging. . . . . . . . . . . . . . . . . 7
1.3 Dissertation Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2 Locating Errors using Value Replacement 17
2.1 Computing Interesting Value Mapping Pairs. . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2 Examples of IVMPs Linked to Faulty Statements. . . . . . . . . . . . . . . . . . . . . . 22
2.2.1 IVMPs at a Faulty Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.2.2 IVMPs Directly Linked to a Faulty Statement . . . . . . . . . . . . . . . . . . 23
2.2.3 IVMPs in the Presence of Erroneously-Omitted Statements . . . . . . . . 25
2.2.4 IVMPs in the Presence of Extraneous Statements . . . . . . . . . . . . . . . . 26
2.3 The Need to Consider Multiple Failing Runs . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.4 Ranking Statements using IVMPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.5 E(cid:11)ectiveness of Value Replacement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.5.1 Setup for Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.5.2 E(cid:11)ectiveness Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.5.3 Experiments with Larger Benchmark Programs. . . . . . . . . . . . . . . . . . 43
2.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3 Value Replacement and Multiple Simultaneous Errors 48
3.1 Techniques to Locate Multiple Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.1.1 Minimal-Computation Technique . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.1.2 Full-Recomputation Technique. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.1.3 Partial-Recomputation Technique . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.2 E(cid:11)ectiveness Comparison of Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
viii
3.2.1 Setup for Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
3.2.2 E(cid:11)ectiveness Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.2.3 Comparison to a Clustering Technique. . . . . . . . . . . . . . . . . . . . . . . . . 63
3.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4 E(cid:14)ciency of Value Replacement 67
4.1 The Cost of Value Replacement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.2 Lossy Techniques to Improve E(cid:14)ciency. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.2.1 Limiting the Number of Statement Instances to Consider . . . . . . . . . . 71
4.2.2 Limiting the Number of Alternate Value Sets to Consider . . . . . . . . . 72
4.2.3 Value Replacement Algorithm with Lossy E(cid:14)ciency Improvements . . 74
4.3 Lossless Techniques to Improve E(cid:14)ciency . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
4.3.1 Removing Redundant Program Execution . . . . . . . . . . . . . . . . . . . . . . 78
4.3.2 Parallelizing the Search for IVMPs . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.4 E(cid:14)ciency Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.4.1 E(cid:14)ciency of Locating Single Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.4.2 E(cid:14)ciency of Locating Multiple Errors . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
5 Locating Memory Errors using Execution Suppression 90
5.1 A Study of Memory Errors and Memory Corruption . . . . . . . . . . . . . . . . . . . 93
5.1.1 Memory Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
5.1.2 Results of the Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
5.1.3 Key Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
5.2 Isolating the First Point of Memory Corruption using Suppression . . . . . . . . 101
5.2.1 Motivational Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
5.2.2 The Suppression Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
5.2.3 Real-world Example using the Suppression Algorithm. . . . . . . . . . . . . 109
5.3 Exposing Program Crashes using Variable Re-ordering. . . . . . . . . . . . . . . . . . 112
5.3.1 The Variable Re-ordering Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 113
5.3.2 Example using the Variable Re-ordering Algorithm. . . . . . . . . . . . . . . 117
5.4 The Complete Execution Suppression Technique. . . . . . . . . . . . . . . . . . . . . . . 118
5.5 Evaluation of Execution Suppression. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
5.5.1 Experiments with Suppression Only. . . . . . . . . . . . . . . . . . . . . . . . . . . 120
5.5.2 Experiments with Suppression and Variable Re-ordering. . . . . . . . . . . 121
5.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
6 Execution Suppression and Multithreading Errors 127
6.1 Locating Multithreading Errors using Suppression . . . . . . . . . . . . . . . . . . . . . 129
6.1.1 Reproducing the E(cid:11)ects of Multithreading Errors . . . . . . . . . . . . . . . . 132
6.1.2 On-the-(cid:13)y Checking for Data Races. . . . . . . . . . . . . . . . . . . . . . . . . . . 134
6.1.3 Performing Suppression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
6.2 Illustrative Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
6.2.1 Example with Harmful Data Race Error . . . . . . . . . . . . . . . . . . . . . . . 137
6.2.2 Example with Non-Data-Race Error . . . . . . . . . . . . . . . . . . . . . . . . . . 141
6.3 Evaluation using Multithreading Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
ix
6.3.1 Setup for Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
6.3.2 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
6.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
7 Suppression Implementation Issues 154
7.1 General Implementation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
7.2 Software-Only Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
7.2.1 Basic Technique for Single-Threaded Programs . . . . . . . . . . . . . . . . . . 156
7.2.2 Generalized Technique for Multithreaded Programs . . . . . . . . . . . . . . 160
7.3 Hardware Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
7.3.1 Using Existing Support for Deferred Exception Handling . . . . . . . . . . 165
7.3.2 Additional Hardware Support through Memory Augmentation. . . . . . 166
7.4 Overhead Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
7.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
8 Towards Correction of Program Errors 170
8.1 Association Rule Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
8.2 BugFix: Automated Assistance for Fixing Errors . . . . . . . . . . . . . . . . . . . . . . 174
8.2.1 Analyzing the Debugging Situation . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
8.2.2 Prioritizing Bug-Fix Suggestions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
8.2.3 Learning from the Debugging Scenario . . . . . . . . . . . . . . . . . . . . . . . . 188
8.3 Case Study. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
8.3.1 Training Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
8.3.2 Encountering New Debugging Situations . . . . . . . . . . . . . . . . . . . . . . . 192
8.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
9 Related Work 197
9.1 Locating Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
9.1.1 Techniques for Locating Errors in General. . . . . . . . . . . . . . . . . . . . . . 197
9.1.2 Techniques for Locating Multithreading Errors . . . . . . . . . . . . . . . . . . 206
9.1.3 Techniques for Locating Speci(cid:12)c Kinds of Errors. . . . . . . . . . . . . . . . . 208
9.2 Fixing Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
9.3 Tolerating Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
9.3.1 Avoiding the Negative E(cid:11)ects of Errors . . . . . . . . . . . . . . . . . . . . . . . . 212
9.3.2 Recovering from the Negative E(cid:11)ects of Errors . . . . . . . . . . . . . . . . . . 213
10 Conclusions 215
10.1 Contributions of this Dissertation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
10.2 Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
Bibliography 223
x