Finding and Tolerating Concurrency Bugs

151 Pages·2013·1.61 MB·English

Finding and Tolerating Concurrency Bugs
by
Jie Yu
A dissertation submitted in partial fulfillment
of the requirements for the degree of
Doctor of Philosophy
(Computer Science and Engineering)
in The University of Michigan
2013
Doctoral Committee:
Assistant Professor Satish Narayanasamy,Chair
Associate Professor Robert Dick
Associate Professor Jason Nelson Flinn
Professor Scott Mahlke
Cristiano Pereira
To my family.
ii
ACKNOWLEDGEMENTS
Firstandforemost,IwouldliketothankmyadvisorProfessorSatishNarayanasamy,
for his endless support during the past five years. It has been an honor to work with
him and to be one of his first batch of PhD students. From him, I have learnt how
to write papers, make presentations, and most importantly, be confident and strong.
Satish has given me the freedom to pursue various projects that match my interests.
He has also provided countless insightful discussions about my research during our
weekly meetings. I appreciate all his contributions of time, ideas, and funding to
make my PhD experience productive and stimulating.
I also would like to thank other members of my PhD dissertation committee,
Professor Robert Dick, Professor Jason N. Flinn, Professor Scott Mahlke and Dr.
Cristiano Pereira. It is no easy task, reviewing a thesis, and I am grateful for their
invaluable comments and constructive criticisms that greatly improved my disserta-
tion.
This dissertation is partially funded by National Science Foundation (NSF) and
Intel Corporation, and I would like to thank both organizations for their generous
support. I would like to thank Dr. Cristiano Pereira and Dr. Gilles Pokam from
Intel for their valuable discussions and insights from industry perspectives.
I also would like to thank all my labmates. I would like to thank Dongyoon Lee
for sharing ideas and brainstorming with me throughout my PhD studies, Abhayen-
dra Singh for clarifying my uncertainties about memory consistency issues in many
iii
occasions, and Chun-Hung Hsiao for working with me on the smartphone project
offering me unremitting assistance. Thanks also go to other members in the lab,
including Shaizeen Aga and Gaurav Chadha, for making the lab such a comfortable
home for me.
My time at Michigan was made enjoyable in large part due to the many friends
and groups that became a part of my life. I would like to thank Lujun Fang and
Yunjing Xu for always hanging out with me, sharing their visions and big ideas.
Special thanks go to Yunjing for taking me to the hospital in midnight when I was
bleeding badly. I would like to thank Lujun Fang and Yudong Gao for being my
roommates for years, making a sweet home for me. Special thanks go to Yudong for
playing basketball with me all the time. Thanks also go to my friends who have not
been mentioned yet, including Junxian Huang, Yi Li, Feng Qian, Li Qian, Zhiyun
Qian, Zhaoguang Wang, Qiang Xu and Xinyu Zhang, for providing support and
friendship that I needed.
Lastly, I would like to thank my family for their unconditional love and support.
My hard-working parents, Jianzu Yu and Shuyan Li, have sacrificed their lives for
me and provided unconditional love and care. I love them so much, and I would not
have made it this far without them. And most of all, I would like to thank my loving,
supportive and encouraging wife Panmei Chen. For this PhD, we have been apart
for almost three years. This dissertation would not be possible without the love and
support from her.
iv
TABLE OF CONTENTS
DEDICATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii
ACKNOWLEDGEMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii
LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x
ABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
CHAPTER
I. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Two Hypotheses and Interleaving Idioms . . . . . . . . . . . . . . . . . . . . 3
1.2 Finding Concurrency Bugs by Exposing Untested Interleavings . . . . . . . . 5
1.3 Tolerating Concurrency Bugs by Avoiding Untested Interleavings . . . . . . 6
1.4 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.5 Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
II. Background and Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.1 Detecting Concurrency Bugs . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.1.1 Data Race Detection . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.1.2 Atomicity Violation Detection . . . . . . . . . . . . . . . . . . . . . 14
2.2 Exposing Concurrency Bugs . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2.1 CoverageDriven Testing . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2.2 Stress Testing and Random Testing . . . . . . . . . . . . . . . . . . 15
2.2.3 Systematic Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2.4 Active Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.5 Test Input Generation . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3 Tolerating Concurrency Bugs . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
III. Encoding Tested Interleavings Using Interleaving Idioms. . . . . . . . . . . 20
3.1 Two Hypotheses about Concurrency Bugs . . . . . . . . . . . . . . . . . . . 21
3.2 Interleaving Idiom . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.3 Canonical Idioms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.4 Relation with Concurrency Bugs . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.5 Empirical Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
IV. Exposing Untested Interleavings: Maple . . . . . . . . . . . . . . . . . . . . . 30
4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
v
4.2 Online Profiling For Predicting iRoots. . . . . . . . . . . . . . . . . . . . . . 34
4.2.1 Notations and Terminology . . . . . . . . . . . . . . . . . . . . . . 34
4.2.2 Naive Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.2.3 Non-Mutex Happens-Before Analysis . . . . . . . . . . . . . . . . . 35
4.2.4 Mutual Exclusion Analysis . . . . . . . . . . . . . . . . . . . . . . . 36
4.2.5 Online Profiling Algorithm . . . . . . . . . . . . . . . . . . . . . . . 38
4.2.6 Predicting iRoots for Compound Idioms . . . . . . . . . . . . . . . 41
4.3 Actively Testing Predicted iRoots . . . . . . . . . . . . . . . . . . . . . . . . 43
4.3.1 A Naive Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.3.2 Non-preemptive and Strict Priority Scheduler . . . . . . . . . . . . 44
4.3.3 Complementary Schedules . . . . . . . . . . . . . . . . . . . . . . . 46
4.3.4 Watch Mode Optimization . . . . . . . . . . . . . . . . . . . . . . . 47
4.3.5 Candidate Arbitration . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.3.6 Dealing with Asynchronous External Events . . . . . . . . . . . . . 50
4.3.7 Compound Idioms . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.3.8 Exposing Pre-conditions . . . . . . . . . . . . . . . . . . . . . . . . 53
4.4 Memoization of iRoots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.5.1 Maple Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.5.2 Usage Scenario 1: Exposing Bugs with Bug Triggering Inputs . . . 56
4.5.3 Usage Scenario 2: Coverage-DrivenTesting . . . . . . . . . . . . . 61
4.5.4 Characteristics of Maple . . . . . . . . . . . . . . . . . . . . . . . . 66
4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
V. Avoiding Untested Interleavings I: PSet . . . . . . . . . . . . . . . . . . . . . 69
5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5.2 Encoding Tested Interleavings . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.2.1 Predecessor Sets (PSets) . . . . . . . . . . . . . . . . . . . . . . . . 73
5.2.2 Effectiveness of PSets in Avoiding Concurrency Bugs . . . . . . . . 75
5.2.3 Deriving and Encoding PSets Constraints . . . . . . . . . . . . . . 78
5.2.4 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
5.3 Enforcing Tested Interleavings . . . . . . . . . . . . . . . . . . . . . . . . . . 79
5.3.1 Detecting and Enforcing PSet Constraints . . . . . . . . . . . . . . 80
5.3.2 Architectural Support . . . . . . . . . . . . . . . . . . . . . . . . . 81
5.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
5.4.1 Bug Avoidance Capability . . . . . . . . . . . . . . . . . . . . . . . 84
5.4.2 Learning PSet Constraints . . . . . . . . . . . . . . . . . . . . . . . 85
5.4.3 PSet Constraint Violations in Bug Free Executions . . . . . . . . . 88
5.4.4 Memory Space Overhead . . . . . . . . . . . . . . . . . . . . . . . 91
5.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
VI. Avoiding Untested Interleavings II: LifeTx . . . . . . . . . . . . . . . . . . . . 94
6.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
6.2 Algorithm for Determining LifeTxes . . . . . . . . . . . . . . . . . . . . . . . 98
6.2.1 LifeguardTransactions(LifeTxes) andProfilingAlgorithmOverview 98
6.2.2 Checking Conflict Serializability for LifeTxes . . . . . . . . . . . . 100
6.2.3 Splitting LifeTxes On a Conflict. . . . . . . . . . . . . . . . . . . . 102
6.2.4 Practical Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
6.2.5 Discussion and Limitations . . . . . . . . . . . . . . . . . . . . . . 106
6.3 Runtime Support for LifeTxes . . . . . . . . . . . . . . . . . . . . . . . . . . 107
6.3.1 LifeTx-Stall Design . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
6.3.2 LifeTx-CS Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
vi
6.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6.4.1 Experimental Setup. . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6.4.2 Learning LifeTxes. . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
6.4.3 Characteristics of LifeTxes . . . . . . . . . . . . . . . . . . . . . . . 116
6.4.4 Bug Avoidance Capability . . . . . . . . . . . . . . . . . . . . . . . 119
6.4.5 Performance Study . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
6.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
VII. Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
VIII. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
BIBLIOGRAPHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
vii
LIST OF FIGURES
Figure
3.1 The canonical idioms for two inter-thread dependencies and two threads. . . . . . . 25
3.2 An idiom1 concurrency bug. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.3 A real idiom4 concurrency bug from MySQL. . . . . . . . . . . . . . . . . . . . . . 27
4.1 Overview of the framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.2 Infeasible iRoots due to non-mutex happens-before relations. . . . . . . . . . . . . 35
4.3 Infeasible iRoots due to mutual exclusion. . . . . . . . . . . . . . . . . . . . . . . . 37
4.4 Predicting iRoots for compound idioms. . . . . . . . . . . . . . . . . . . . . . . . . 42
4.5 The ideal situation for exposing an idiom1 iRoot A⇒B. . . . . . . . . . . . . . . 44
4.6 The naive approach could deadlock when exposing an idiom1 iRoot A⇒B. . . . . 44
4.7 The situation in which the watch mode is turned on for exposing an idiom1 iRoot
A⇒B. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.8 Problem with asynchronous external events. . . . . . . . . . . . . . . . . . . . . . . 51
4.9 Expose a compound idiom iRoot A⇒B...C ⇒D. . . . . . . . . . . . . . . . . . . 51
4.10 A pre-condition exists when trying to expose iRoot A⇒B. . . . . . . . . . . . . . 51
4.11 Comparison with different testing methods using the same amount of time. M
standsforMaple,PstandsforPCT,LstandsforPCTLarge,DstandsforRandDelay,
and C stands for CHESS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.12 Comparisonwith different testing methods. X-axis is the number of test runs,and
Y-axis is the total number of iRoots exposed. . . . . . . . . . . . . . . . . . . . . . 65
4.13 Memoizationsavestestingtime. Yaxisisnormalizedtotheexecutiontimewithout
memoization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5.1 PSet constraints for an interleaving. . . . . . . . . . . . . . . . . . . . . . . . . . . 74
5.2 A data race bug in Mozilla (Mozilla-7 in Table 3.5). . . . . . . . . . . . . . . . . . 75
5.3 An atomicity violation bug in Mozilla [43] (Mozilla-1 in Table 3.5). . . . . . . . . . 76
viii
5.4 An atomicity violation bug in Mozilla, which will not raise an AVIO [43] invariant
violation. (Mozilla-4 in in Table 3.5). . . . . . . . . . . . . . . . . . . . . . . . . . . 77
5.5 Order violation bug in Mozilla, which is neither a data race nor an atomicity vio-
lation. (Mozilla-9 in Table 3.5). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
5.6 Format for encoding an instruction’s PSet. . . . . . . . . . . . . . . . . . . . . . . . 79
5.7 Number of test runs required for learning PSets and AVIO invariants. . . . . . . . 89
5.8 Proportion of static memory instructions with a particular PSet size (normalized
to the total number of static memory instructions in the application binary and
libraries that were executed in at least one test run). . . . . . . . . . . . . . . . . . 91
6.1 A conflict serializability violation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
6.2 A conflict-serializability violation detected across multiple semantic-segments. . . . 104
6.3 Conflicts due to memory operations executed in synchronization functions. . . . . . 105
6.4 Number of test runs required for getting stable cutpoints. . . . . . . . . . . . . . . 117
6.5 Footprint distribution (dynamic). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
6.6 Length distribution (dynamic). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
6.7 LifeTx-Stall performance overhead. . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
ix
LIST OF TABLES
Table
3.1 Brief descriptions about concurrency bugs that we used in our studies. . . . . . . . 28
3.2 Empirical study on 34 documented bugs. . . . . . . . . . . . . . . . . . . . . . . . . 29
4.1 Bug exposing capability given bug triggering inputs. All the time reported in the
table are in seconds. TO stands for timeout (24 hours). In the type column, S
stands for synthetic bugs, E stands for extracted bugs, R-K stands for real bugs
which are known, and R-U stands for real bugs which are unknown. * The root
causeof Bug #12andbug Bug #13havenotbeenconfirmedyet. Theyareexposed
when attempting idiom1 iRoots.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.2 Memoizationhelpexposebugs morequickly. Allthe time reportedinthe table are
in seconds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.3 Runtime overheadof Maple comparing to native execution time. . . . . . . . . . . 66
4.4 The success rate of the active scheduler (# successfully exposed iRoots / # total
predicted iRoots). For apache and mysql, we experimented with 100 randomly
selected candidate iRoots. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.1 Bugdetectioncapability. ComparingPSetwithahappens-beforedataracedetector
and AVIO [43]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.2 AvoidingbugsusingPSetconstraints. True constraintviolationsarerelatedto the
bug. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
5.3 PSet constraint violations in bug-free executions. . . . . . . . . . . . . . . . . . . . 88
5.4 Binary Size Increase. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
6.1 Baseline configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
6.2 Bug avoidance capability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
6.3 Characteristics of critical LifeTxes. . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
6.4 Runtime statistics for (a) LifeTx-Stall, (b) LifeTx-CS. . . . . . . . . . . . . . . . . 123
6.5 Granularity of conflict detection: Block vs Word. . . . . . . . . . . . . . . . . . . . 124
x