Types for Correct Concurrent API Usage - scs technical report

253 Pages·2011·1.33 MB·English

Types for Correct Concurrent API Usage
Nels E. Beckman
CMU-ISR-10-131
December 2010
InstituteforSoftwareResearch
SchoolofComputerScience
CarnegieMellonUniversity
Pittsburgh,PA15213
ThesisCommittee:
JonathanAldrich(Chair)
StephenBrookes
WilliamScherlis
SriramRajamani(MicrosoftResearchIndia)
Submittedinpartialfulfillmentoftherequirements
forthedegreeofDoctorofPhilosophy.
Copyright⃝c 2010NelsE.Beckman
This work was supported in part by DARPA grant #HR0011-0710019, NSF grant CCF-0811592, R&D Project
AeminiumCMU-PT/SE/0038/2008intheCMU—Portugalprogram,ArmyResearchOfcegrant#DAAD19-02-1-
0389entitled“PerpetuallyAvailableandSecureInformationSystems,”theDepartmentofDefense,andtheSoftware
IndustryCenteratCMUanditssponsors,especiallytheAlfredP.SloanFoundation. Theauthorwassupportedby
aNationalScienceFoundationGraduateResearchFellowship(DGE-0234630).
Keywords: API, object protocol, typestate, concurrency, multi-threading, object-oriented,
type theory, type system, static analysis, specification, verification, Java, empirical, polymor-
phism,inference,probabilistic,transactionalmemory,STM,optimization
ForDr. BarbaraBeckman,agreatscientistandawonderfulmother
iv
Abstract
This thesis represents an attempt to improve the state of the art in our ability to
understandandcheckobjectprotocols,withaparticularemphasisonconcurrentpro-
grams. ObjectprotocolsarethepatternsofuseimposedonclientsofAPIsinobject-
oriented programs. We show through an empirical study of open-source object-
oriented programs that object protocols are quite common. We then present “Sync-
or-Swim,”amethodologyandsuiteofaccompanyingtoolsforcheckingatcompile-
time that object protocols are used and implemented correctly. This methodology is
based upon the existing access permissions method of alias control, which is here
extended to be sound in the face of shared-memory concurrency. The analysis is
formalized as a type system for an object-oriented calculus, and then proven to be
free from false-negatives using a proof of type safety. The type system is extended
withparametricpolymorphism,or“generics,”inordertoincreaseitsabilitytocheck
commonly occurring patterns. An implementation of the approach, a static analysis
for programs written in the Java programming language, is presented. This imple-
mentation was used to perform a series of case studies whose goal was to evaluate
the ease of use, expressiveness and ability to verify commonly occurring patterns.
These case studies are presented. Next, an approach and an associated tool for in-
ferring access permission annotations is presented. This inference tool can reduce
the burden of using our protocol-checking approach by automatically inferring the
required typing annotations. This inference is built upon a system of probabilistic
constraints,whichallowstheeasyencodingofheuristics. Finally,anoptimizationof
software transactional memory runtimes is presented. This optimization is enabled
by the typing annotations required to use the concurrent protocol checker and can
remove some of the overhead typically associated with transactional memory sys-
tems. As a result of the work presented in this thesis, it is possible to guarantee the
absence of certain API usage errors even in concurrent programs, and to do so with
a low burden on programmers. By adhering to such an approach, programmers can
producemorereliablesoftware.
vi
Acknowledgements
TheprocessofgettingaPh.D.isalongone,andatryingone. Duringthepastfive
yearsIhaveexperiencedinnumerablesetbacks. Eachonedashedmyself-confidence
in some small way, and taken together might be considered to be quite discourag-
ing. But through the setbacks I experienced true and powerful transformation, both
intellectually and personally. Critically, such a journey would have been unbearable
wereitnotformyamazingcolleagues,mywisefacultyadvisors,andthesupportof
myfriendsandfamily.
CarnegieMellonisanamazingplacetostudycomputerscience,anditislargely
greatbecauseofthehighqualityofitsstudents. Ihavelearnedalotfromthestudents
in the Principles of Programming group, our resident programming language theo-
rists. Before arriving at CMU, type theory and formal logic were foreign concepts
to me, and without their patient explanations, particularly those of Rob Simmons,
William Lovas and Tom Murphy VII, I never would have had a chance to see the
beautythatunderlieseverydayprogramming.
I was lucky enough to play an advisor role for several talented students, Yoon
PhilKim,DuriKim,andPaulRichardson. Youwereallquicklearnersandexceeded
my expectations. Some of your work is described in these chapters! I hope you are
allhappywiththeresults. Anyerrorsare,ofcourse,myown.
IamalsogratefulforalltheothersoftwareengineeringPh.D.studentsintheIn-
stitute for Software Research. They are an entertaining and intelligent bunch. Make
sure you keep the SSSGs well-stocked with treats and lively debate after I am gone.
To those students in the ISR who have gone before me, particularly George Fair-
banks and Shang-Wen “Owen” Cheng, thanks for all of the useful advice. It was
invaluable!
A special thanks goes out to the members of the PLAID research group. The
PLAID group is more like a family than a research group. Its members are enter-
taining and kind, and there is always a lively and contrarian discussion going on
amongstitsmembers. IhavespentaparticularlylargeamountoftimewithThomas
LaToza, Joshua Sunshine, Donna Malayeri, Sven Stork and Ciera Jaspan. We have
eatenlunchtogetherincalculabletimes. Sorryforalwaysforgettingtobringmoney.
I will pay you back soon! We have also attended a number of conferences together,
and I can say that these have been some of the most fun and rewarding experiences
of my career. Thanks a lot guys. Ciera and I started graduate school at the same
time, and we have spent a lot of time eating and working together. And now we are
going to graduate just about at the same time. Congratulations, and best of luck to
youCiera! Iamexcitedtohearabouttheprogressionofyouracademiccareer.
An extra bit of thanks goes out to Kevin Bierhoff. In addition to being a good
friend,Kevinhasbeenanexcellentcollaborator. Myworkhaslargelycontinuedthe
work that Kevin started and as such, Kevin and I have worked very closely. Kevin,
thanksforbeingsuchagreatfriendandforgivingmesomanygoodideas!
I am very appreciative of my entire thesis committee for generously giving their
time to me. All professors are enormously busy, which is why I have been so im-
pressed by the time my committee members, Bill Scherlis, Stephen Brookes, and
Sriram Rajamani, have given to meeting with me and reading my work. Their feed-
backhasbeeninvaluable.
While not on my committee, Aditya Nori has been a sort of second advisor to
me. MytwotripstoIndiatovisitMicrosoftResearchhavebeentransformative,both
from an intellectual perspective and from a wider cultural perspective. Aditya has
been a big part of these experiences, and since my first trip we have stayed in close
contact, sharing ideas and research progress. It was great getting the opportunity to
workwithhimagainonaprojectrelevanttomythesis!
And of course I am extremely indebted to Jonathan Aldrich, my thesis advisor
and mentor. Jonathan is a great advisor. He is intelligent, naturally, but intelligence
for Carnegie Mellon professors is merely par for the course. He is also patient, firm
andencouraging,andwieldsthosecharacteristicsasthesituationmandates. Iamnot
sureIcouldhavegottenthroughthisprocesswithanyoneelse. ThankyouJonathan!
When asked to give advice on a successful graduate experience, I have always
stressed that one cannot make it through a Ph.D. program without a strong network
of friends. This is absolutely true. Fortunately for me, I have a number of great
friends and an amazing, supportive family, without whom I never would have made
itthroughthisjourney.
IhavealwaysbeengratefulformyfriendsfromNewOrleanslikeRandVoorhies
and Nick Perrin, along with my West Coast friends, including Jeff Dralla and Greg
Mooney. I am sure you guys have all wondered what I have been up to for the past
fiveyears. Well,hereyougo.
Pittsburgh also has been a great place to meet interesting people. The list of
amazing people that I have met here is too numerous to fit in this document, but I
would like to especially thank (in no particular order) Emily Keebler, Mike Cronin,
SuzanneWeesies,MikeTschantz,JasonFranklin,RyanKelly,JessicaNelson,Reed
Taylor, Laura Halderman, Erika Laing and some of the great friends who have left
PittsburghlikeMajaH.Ahmetovic,LauraHiatt,StephenMagill,MikeDinitz,Chris
Martin, Marcus Louie and Cortney Higgins. I am all too aware that I have left out
someonewhohasbeenverykindtomehere,andforthatIamreallysorry!
Thankstomyawesomefamilyforgivingmesomuchsupportandguidance. My
extended family has always been so great. Tom and Holly, Ira and Jay, and Jane,
thank-you all. And those who are no longer around to see this day, Jim, Donnis
and Ed, I really miss you. Kristin, you are an awesome sister and I am so happy
for your success! Mom and Dad, you guys are the best, and you instilled in me the
confidence,curiosityanddisciplineittakestobesuccessful. Thankyousomuch.
Finally,Brianne,yourlovehasmeantsomuchtome. Youareakind,intelligent
and beautiful woman, and I am so glad we will be spending the rest of our lives
together. I will always remember graduate school fondly because, at the very least,
itbroughtustogether. Iloveyou!
My advice to future graduate students? Havefun, make friends, fall in love, and
learnasmuchasyoupossiblycan.
viii
Contents
1 Introduction 1
1.1 ConcurrencyandObjectProtocols . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Overview: ToSync,ortoSwim? . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3.1 DoProtocolsReallyMatter? . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3.2 CheckingProtocolstheSync-or-SwimWay . . . . . . . . . . . . . . . . 5
1.3.3 OneMoreDeveloperResponsibility? . . . . . . . . . . . . . . . . . . . 6
1.4 ThisThesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.4.1 ThesisStatement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.4.2 Hypotheses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.5 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.6 PotentialImpact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.7 ThesisOutline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2 AnEmpiricalStudyofObjectProtocolsintheWild 13
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2.1 DefinitionsandScope . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2.2 ExperimentalProcedure . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.3 ProgramsUnderAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.2.4 Risks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.3.1 ProtocolDefinitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.3.2 ProtocolCategories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.3.3 ProtocolUsage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.4.1 SanityCheck . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.4.2 WidelyUsedProtocols . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.4.3 ProtocolCategories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.4.4 OtherObservations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.5 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.6 FutureWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
ix
3 Approach: ATypeSystemforCorrectConcurrentAPIUsage 37
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.2.1 ObjectProtocolModeling . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.2.2 AccessPermissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.2.3 TrackingHeldLocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.2.4 VerifyingOurExamples . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.3 Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.3.1 BasicLanguageSyntax . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.3.2 PermissionSyntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.3.3 PermissionManipulationandWell-Formedness . . . . . . . . . . . . . . 57
3.3.4 Type-CheckingandProgramWell-Formedness . . . . . . . . . . . . . . 61
3.3.5 TransactionalMemoryandAtomicBlocks . . . . . . . . . . . . . . . . 69
3.4 TheSync-or-SwimChecker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.5 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.5.1 VerifyingBehaviorofConcurrentPrograms. . . . . . . . . . . . . . . . 72
3.5.2 RaceDetection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
3.6 FutureWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
3.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4 ProofofSoundness 79
4.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
4.2 LanguageDefinition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.2.1 LanguageDifferencesandSimplifications . . . . . . . . . . . . . . . . . 81
4.2.2 Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.2.3 PermissionSyntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
4.2.4 PermissionWell-FormednessandManipulation . . . . . . . . . . . . . . 85
4.2.5 Type-CheckingandProgramWell-Formedness . . . . . . . . . . . . . . 87
4.2.6 DynamicSemantics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
4.2.7 ProofJudgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
4.3 TheoremsandProofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
4.3.1 Top-LevelProofofSafety . . . . . . . . . . . . . . . . . . . . . . . . . 103
4.3.2 Single-ThreadedGuarantees . . . . . . . . . . . . . . . . . . . . . . . . 107
4.3.3 Thread-LevelProofofSafety . . . . . . . . . . . . . . . . . . . . . . . 108
5 PolymorphicAccessPermissions 111
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
5.3 PolymorphicAccessPermissions . . . . . . . . . . . . . . . . . . . . . . . . . . 115
5.3.1 TheSyntaxofPermissionsandAbstraction . . . . . . . . . . . . . . . . 116
5.3.2 StaticSemanticsforPermissionsAbstractionandApplication . . . . . . 118
5.3.3 AbstractingOverQuantificationClassifiers . . . . . . . . . . . . . . . . 121
5.3.4 QuantifyingOverSymmetricPermissionKinds . . . . . . . . . . . . . . 122
5.3.5 TypingRules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
x