ebook img

Verifying Concurrent Stacks by Divergence-Sensitive Bisimulation PDF

0.29 MB·
Save to my drive
Quick download
Download
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.

Preview Verifying Concurrent Stacks by Divergence-Sensitive Bisimulation

(cid:73) Verifying Concurrent Stacks by Divergence-Sensitive Bisimulation XiaoxiaoYanga,Joost-PieterKatoenb,HuiminLina,HaoWub aStateKeyLaboratoryofComputerScience,InstituteofSoftware,ChineseAcademyofSciences bSoftwareModelingandVerification,RWTHAachenUniversity,Germany Abstract Theverificationoflinearizability–akeycorrectnesscriterionforconcurrentobjects–isbasedontracerefinementwhosechecking isPSPACE-complete. Thispapersuggeststousebranchingbisimulationinstead. Ourapproachisbasedoncomparinganabstract specificationinwhichobjectmethodsareexecutedatomicallytoarealobjectprogram. Exploitingdivergencesensitivity,thisalso appliestoprogresspropertiessuchaslock-freedom.Theseresultsenabletheuseofpolynomial-timedivergence-sensitivebranching 7 1bisimulationcheckingtechniquesforverifyinglinearizabilityandprogress. Weconductedtheexperimentonthelock-freestacks 0tovalidatetheefficiencyandeffectivenessofourmethods. 2 Keywords: n Linearizability,ProgressProperties,Bisimulation,Divergence,Concurrency a J 1 21. Introduction specification if and only if the set of execution traces of the implementation is a subset of those of the specification. Al- ] Linearizability is a standard correctness condition for the thoughthisapproachcanbeusedwelltoverifylinearizability, L implementation of concurrent objects [1]. Checking lineariz- it is not suitable for verifying progress properties. For exam- P ability amounts to verify that a concurrent object is imple- ple, [8] verifies linearizability based on refinement checking . smented correctly by establishing a certain correspondence be- of finite-state systems specified as concurrent processes with c [tween the object and its sequential specification. In a multi- shared variables. However, that refinement relation cannot be threadedenvironment,however,wealsoexpectthatconcurrent used to prove progress properties. By the standard definition 1 systems satisfy liveness properties [2], which guarantee that of linearizability [1], a sequential object specification is em- v 4certain ”good” events can eventually happen. This includes ployed as the abstract specification, to which any implemen- 0progress properties [12] such as lock-freedom, wait-freedom tation should conform. However, the sequential specification 1and obstruction-freedom, which are introduced to capture the cannot specify liveness. To remedy this, [7] proposes a gen- 6essence of the progress guarantee of non-blocking concurrent eralisationoflinearizabilitythatcanspecifylivenesswhile[6] 0 algorithms[3]. Thesealgorithmsemployfine-grainedsynchro- usesatomicblockstodescribetheabstractspecificationwhere . 1nisations instead of mutex-locks to provide high-performance each method body of an object method should be atomic. Al- 0concurrent implementations of objects, such as Michael-Scott though [6, 7] discuss progress properties of objects based on 7 lock-free queue, lock-free stacks [13, 16] and Harris lock-free sometermination-sensitivecontextualrefinement,theyneedto 1 :list(seethejava.util.concurrentpackage). Nevertheless, definedifferentrefinementrelationsforprogressproperties. vtheprogresspropertyofanobjectcanaffecttheprogressofthe i Instead of the refinement relation between two objects with Xexecutionofaclientprogramthatusestheobject. AsFilipovie´ differentgranularitiesofanatomicoperation,analternativeper- et al. point out ”programmers expect that the behaviour of r spectiveistocapturetherelationshipbymeansofappropriate atheirprogramremainsthesameregardlessofwhethertheyuse equivalencerelation. Thisisjustifiedbythefactthatclientpro- highlyefficientdatastructuresorless-optimizedbutobviously grams expect that the observable behaviour of concrete object correctdatastructures.[11]”Foroptimaluseofconcurrentob- programsisequivalenttothatofabstractones. Themainissue jects,itisessentialthatthebasicpropertiesofobjectsinvolving isthatwhetherthereexistsaproperequivalencerelationforthe linearizability and progress properties are ensured. This em- objectimplementation,andifthereexists,whatkindofequiv- phasizestheimportanceofdeveloping(automated)techniques alence is needed. On the one side, it needs to abstract from forverifyinglinearizabilityandprogressproperties. internal steps of concrete objects as they typically contain im- The main work on verifying linearizability of concurrent plementationdetailsthatarenotpresentinabstractobjects. On objects is using trace refinement checking [5, 8, 10, 11, 21]. theotherside,weexpectthatsuchequivalencepreservessome This means that a concrete implementation refines an abstract desiredpropertiessuchthatforequivalentabstractandconcrete programs,thepropertiescanbecheckedonthe–simpler–ab- stractprogramratherthanontheconcreteprogram. (cid:73)ThisresearchwassupportedbyNSFC61100063andbyaHumboldtFel- lowship(X.Y.)fromAlexandervonHumboldtFoundation. We are therefore motivated to explore the equivalence rela- January24,2017 tionontheexternallybehaviorofobjects,andinvestigateacon- 2. Preliminaries venient and efficient method to verify both linearizability and progresspropertiesofconcurrentobjects. Ourmethodemploys 2.1. AbstractandConcreteObjects atailoredversionofbranchingbisimulation[17,18,19]toan- Weusetwokindsofdescriptionsforconcurrentobjects: ab- alyzethebehaviourofconcurrentobjects. Branchingbisimula- stractandconcrete. Abstractobjectscanberegardedasacon- tion is an action-based version of stutter bisimulation [18, 19] currentspecification,whereeachmethodbodyofeveryobject that basically allows to abstract from sequences of internal methodisasingleatomicoperation. Concreteobjectsinvolves steps.Thisideawasfirstproposedinourtechnicalreport[9].In more intricate interleavings that refines the implementation of thispaper,wepresenttheinspirationandmotivationoftheidea abstractobjectsbyreplacingthesingleatomicoperationofeach in [9], and shows the verification method on the non-blocking methodbodywithfine-grainedlock(orlock-free)synchroniza- stacksindetailbydirectlyestablishingthedivergence-sensitive tiontechnique. Forabstractandconcreteobjects,eachmethod branchingbisimulationbetweenthestackanditsspecification. startswithamethodcall,andendswithareturnaction. Ourmaincontributionsaresummarizedasfollows: Forinstance,anabstractcounterwithmethodinc()isgiven in Figure 1 (i), where t is a local variable and c a shared vari- able,statementsembracedbyatomic{}isanatomicoperation. • Weusetheatomicblocktospecifyanabstractobjectpro- A concrete counter with the CAS instruction is given in Fig- gram.Weobservethattheeffectofthemethodcallofcon- ure1(ii). TheCASinstructionallowssynchronizationbetween crete(fine-grained)objectscanbeabstractedintothehigh- threadsatafinergrain. levelatomicblockoftheabstractobjectprogramsuchthat the fine-grained implementation takes the same effect as theabstractatomicoperation. Sincetheeffectsofmethod (i) int inc() (ii) int inc() calls can be equivalently characterized by the observable atomic {t:=c; do t := c; actionscallandreturn,weshowthatbranchingbisimula- c:=t+1;} while(CAS(c,t,t+1)) return; return; tionisaprecisenotiontocapturetheexternallyobservable equivalencebetweentheconcrete(fine-grained)objectand theabstract(atomic)object. Wefurthershowthatbranch- Figure1: Abstractcounter(i)andconcretecounter(ii). ingbisimulationpreserveslinearizability. 2.2. Lock-FreedomandWait-Freedom • To verifyprogress properties, we explorethe divergence- sensitiveequivalencefortheabstractandconcreteobjects. Lock-freeandwait-freepropertiesarenon-blockingprogress Divergence-sensitive branching bisimulation implies the conditions. Informally,anexecutionofobjectmethodsislock- preservationofLTL(cid:13)(thenext-freeLTL)equivalence,so freeifitguaranteesthatsomethreadcancompleteanystarted itpreservesvariousdesirableprogressproperties. operationontheshareddatastructureinafinitenumberofsteps [12]. Anexecutioniswait-freeifitguaranteesthateverythread willcompleteanoperationinafinitenumberofsteps[12]. • Insteadoftracerefinementtechniquewhichiscomputedin Suppose that the counter object provides a unique method PSPACE-complete,ourmethodenablestheuseofbranch- inc()forincrementingasharedvariablec. Figure1(i)showsa ingbisimulationcheckingtechniquesandtoolsforproving wait-freeimplementationinwhichtheincrementisdoneatomi- correctness and progress of objects, which can be com- cally.Figure1(ii)presentsalock-freeimplementationbyusing putedinpolynomialtimeforfinitesystems. theCASinstruction. • We successfully apply branching bisimulation with ex- Example1. Considerthefollowingclientprogram: plicit divergence to verify linearizability and lock-free while (1) do inc(); || inc(); print(z:=0); property of non-blocking stacks [16, 27], and reveal a new bug violating the lock-freedom in the revised stack where||denotestheparallelcomposition. Iftheclientemploys [30]. Thus divergence-sensitive branching bisimulation the wait-free counter in Figure 1 (i), then the right thread can provides us a unified framework to verify linearizability always terminate and prints z:=0 since the wait-freedom of while preserving progress properties that are specified in inc() guarantees the termination. But if the client uses the LTL . lock-freecounterinFigure1(ii),theexecutionofinc()bythe (cid:13) rightthreadmaybenon-terminating,sincetheCASinstruction in Figure 1 (ii) may continuously fail such that the left thread Overview. Section2definestheobjectsystem.Section3intro- cancontinuouslyupdatesharedvariablec.Thustherightthread duceslinearizabilityandthetracerefinement. Section4shows neverprintsz:=0. (cid:117)(cid:116) branching bisimulation for concurrent objects. Section 5 ex- ploresthedivergence-sensitivebranchingbisimulationtoverify Inpractice,toachievethedesiredprogresspropertiesofcon- progressproperties. Section6conductstheexperimentonveri- currentobjects,thegarbagecollectionmechanismwithaproper fyingthenon-blockingstacks. Section7makestheconclusion. progressconditionalsoneedstobeconsidered. 2 Example2. Consider the concurrent stack with hazard point- Inthecontext,wesometimesuse∆todenotean(abstractor ers [27], where function Retire Node reclaims the released concrete)objectsystemorthecorrespondingobjectprogram. nodestoavoidtheABA-error. Toguaranteethelock-freeprop- Eachpathoftheobjectsystemisanexecutionoftheobject. ertyofthestack, thegarbagecollector Retire Nodeneedsto Anexecutionfragmentρ(alsocalledapathfragment)isafinite bewait-free. However,if popinvokesanothergarbagecollec- orinfinitealternatingsequenceofstatesandactionsstartingin torin[30],whichisnotwait-freeandnotlock-free,thenitwill theinitialstates0thatisρ={s0α0s1α1··· |(si,αi,si+1)∈ −−→}, resultintheunexpectednon-terminatingexecutionofpopsuch where α ∈ Act,s = ς. A trace is a finite or infinite sequence i 0 thattheimplementationoftheconcurrentstackin[30]violates ofobservableactionsobtainedfromapathbyabstractingaway thelock-freeproperty. ThedetailisshowninSection6. (cid:117)(cid:116) thestatesandτ-transitions. Weshallwrites→−a s(cid:48)toabbreviate (s,a,s(cid:48)) ∈−→. A state s(cid:48) is reachable if there exists a finite 2.3. ObjectSystems executionfragmentsuchthats0 −−α→0 s1 −−α→1 ···−−α→n sn = s(cid:48). Foranalysingconcurrentobjects,weareinterestedintheim- plementation of objects (i.e., object internal actions) and the 3. Linearizability possible interactions (i.e., call and return) between the client andtheobject. Forclients,objectactionsareinternalandusu- We briefly introduce the standard linearizability definition allyregardedasinvisibleactions,denotedbyτ. [1]. We use a history H, which is a finite sequence of ac- LetCbeafixedcollectionofobjects.Wehaveasetofactions tions call and return, to model an execution of an object sys- Actintheformof: tem. Givenanobjectsystem∆,itssetofhistoriesisdenotedby Act::=τ|(t,call,o.m(n))|(t,ret(n(cid:48)),o.m) H(∆).SubhistoryH |tofHisthesubsequenceofallactionsin H whosethreadnameist. Thekeyideabehindlinearizability whereo∈Candtisthethreadidentifer.Silentactionτisinvis- istocompareconcurrenthistoriestosequentialhistories. ibleandtheothertwoactionsarevisible,where(t,call,o.m(n)) Ahistoryissequentialif(1)itstartswithamethodcall,(2) means thread t invokes method call method m(n) of object o calls and returns alternate in the history, and (3) each return withparametern;and(t,ret(n),o.m)meansthreadtreturnsthe valuen(cid:48)formethodmofobjecto. matches immediately the preceding method call. A sequential history is legal if it respects the sequential specificationof the We assume there is a underlying programming language to object. A call is pending if it is not followed by a matching describe the object program. The language equipped with the return. Letcomplete(H)denotethehistoryobtainedfromHby operational semantics can generate a state transition system, deletingallpendingcallsifthemethodcallhasnottakeneffect whichiscalledtheobjectsystem. Asin[7,5], togeneratethe yetoraddingthecorrespondingreturnactionifthemethodcall object behaviour, we assume the most general client that re- hastakeneffect. peatedlyinvokestheobject’smethodsinanyorderandwithall Weusee.callande.rettodenote,respectively,theinvocation possibleparameters. and response events of an operation e. An irreflexive partial Definition2.1(Objectsystems). LetCbeafixedcollectionof order <H on the operations is defined as: (e,e(cid:48)) ∈ <H if e.ret objects. An object system ∆ is a labelled transition system precedese(cid:48).callinH. Operationsthatarenotrelatedby<H are (S,−→,Act,ς),where saidtobeconcurrent (oroverlapping). If H issequentialthen < is a total order. We first define the linearizability relation H • S isthesetofstates; betweenhistories. • Act = {τ, (t,call, o.m(n)), (t,ret(n(cid:48)), o.m) | o ∈ C,t ∈ {1...k}}, where k is the number of threads, Act is the set Definition3.1(Linearizabilityrelationbetweenhistories). ofactions; H (cid:118) S,read“H islinearizablew.r.t. S”,if(1)S issequen- lin • −→⊆S ×Act×S isthetransitionrelation; tial,(2)H|t=S|tforeachthreadt,and(3)<H ⊆ <S. (cid:117)(cid:116) • ς ∈S istheinitialstate. Thus H (cid:118) S ifS isapermutationof H preserving(1)the lin Further,wehavethefollowing: order of actions in each thread, and (2) the non-overlapping method calls in H. Let Γ denote the sequential specification • Abstract object system, denoted by Θ, is an object sys- andH(Γ)thesetofallhistoriesofΓ. ForeachS ∈ H(Γ),S is temofanabstractobject,wherethemethodbodyofeach alegalsequentialhistory. Thelinearizableobjectisdefinedas methodisspecifiedbyanatomicoperationatomic{}. follows. • Concrete object system, denoted by O, is an object sys- tem of a concrete object, where the method body of Definition3.2(Linearizabilityofobjectsystems). An object eachmethodisspecifiedbythelowlevelsynchronization system ∆ is linearizable w.r.t. a sequential specification Γ, if mechanism. ∀H1 ∈H(∆). (∃S ∈H(Γ).complete(H1)(cid:118)lin S). (cid:117)(cid:116) • Linearizablespecification:foraconcreteobjectsystemO, Linearizability can be casted as the trace refinement [5, 8]. weuseanabstractobjectsystembyturningeachmethod Tracerefinementisasubsetrelationshipbetweentracesoftwo bodyofOintoanatomicblock[6,8],asthecorresponding objectsystems,animplementationandalinearizablespecifica- specification,whichiscalledlinearizablespecification. tion. Lettrace(∆)denotethesetofalltracesin∆. Formally,for 3 twoobjectsystems∆1 and∆2,wesay∆1 refines∆2,writtenas wheres2==⇒τ lkdenotesstatelkisreachablefroms2byperform- ∆ (cid:118) ∆ ,ifandonlyif trace(∆ )⊆trace(∆ ). ingzeroormoreτ-transitions. ∆ and∆ arecalledbranching 1 tr 2 1 2 1 2 bisimilar,denoted∆ ∼ ∆ ,ifthereexistsabranchingbisim- 1 B 2 Theorem3.3. [8]Let∆beanobjectsystemandΘthecorre- ulationRsuchthat(ς ,ς )∈R. (cid:117)(cid:116) 1 2 spondingspecification. Allhistoriesof∆arelinearizableifand onlyif∆(cid:118)tr Θ. Notethatthedefinitionof∼Bcanalternativelybegivenby: (cid:91) Theabovetheoremshowsthattracerefinementexactlycap- ∼B= {R|Risabranchingbisimulationwith(ς1,ς2)∈R} tureslinearizability. Aproofofthisresultcanbefoundin[8]. By standard means, we can prove that ∼ is an equivalence Remark: Abstractobjectswiththehigh-levelconstruct,i.e., B relation,i.e.,∼ isreflexive,symmetricandtransitive. atomic blocks, provide us a concurrent specification [6, 7, 8]. B Relatingtotheimplementationofconcurrentobjects,thein- Usingasequentialspecificationstandardforlinearizability,itis tuitivemeaningofDefinition4.1isexplainedasfollows. Con- necessarytomaptheconcurrentexecutionstosequentialones dition (1) states that every outgoing transition of s labelled limitingourreasoningtothesesequentialexecutions[14]. The 1 with a visible action must be matched by an outgoing transi- permutationofconcurrentexecutionstosequentialonesisnot tion of s with the same visible action. Condition (2) has two convenient and even some highly-optimized objects, such as 2 cases: (i) for each transition s → s(cid:48) labelled with τ, either the collision array in the elimination stack [13], do not have 1 1 (s(cid:48),s )∈R,whichmeansthestepfroms tos(cid:48) isastutterstep theintuitivesequentialsequencescorrespondingtothem. Con- 1 2 1 1 that does not change the state of the shared object; (ii) or the currentspecificationhasamoredirectandnaturalrelationship transition is matched by a path fragment s l ...l s(cid:48) such that withtheconcurrentimplementation. Itallowsmethodcallsnot 2 1 k 2 (s(cid:48),s(cid:48)) ∈ Rand s l l ...l areaseriesofstutterstepsandfor toterminate(e.g.,neverreturn)andtheexecutionofmethodsto 1 2 2 1 2 k each l, (s ,l) ∈ R (a path is written as s s ...s for short). overlapwiththeexecutionsofothers. Itadequatelymodelsthat i 1 i 0 1 n Note that stutter steps from s to l do not change the object any method non-termination and overlapping execution inter- 2 k to a new state, but the last step from l to s(cid:48) is an atomic step valintheimplementation(i.e.,concreteobjectsystems)canbe k 2 at which point the entire effect of the method call takes place, reproducibleinthespecification(i.e.,abstractobjectsystems). which corresponds to the execution of the atomic block in the abstractobject. Afterthestepthattakeseffect,themethodcall 4. BranchingBisimulationforConcurrentObjects maycontinuetotakezeroormorestutterstepstocompletethe remaining method call. Since R is symmetric, the symmetric Byobservingtheexternalbehaviorcallandreturn,whenthe counterpartsofcases(1)-(2)alsoapply. abstract object system Θ performs an operation in an atomic As we see, Definition 4.1 reveals an equivalence relation step (t,τ) that takes effect for the abstract stack, the concrete ∼ between(concurrent)abstractobjectΘandconcreteobject B oneOmaytakeseveralτ-stepstocompletetheoperationwith O, which shifts the linear-time paradigm to a branching-time thesameeffectastheatomicoperationofΘ. Thisphenomenon paradigm. The key point is the condition (2). It is easy to un- resultsinthattheatomicstepofΘcanbemimickedbyapath derstandtheformercasethatΘsimulatesO. Forthelattercase withstutterstepsofOandastepthattakeseffectandthesestut- thatOsimulatesΘ,condition2-(ii)(k ≥ 0)inDefinition4.1is terstepsareexactlyrighttheimplementationdetailsofthecon- required, whichprovidesthatforeachstateinΘ, therecanbe creteobject, whichdonotchangethestateoftheobject. Vice matchedbyastuttersequenceofτ-stepsinO. versa,apathfragmentofconcreteobjectsystemOcanalsobe mimickedbytheatomicstepofΘ. Suchthecorrespondingre- Theorem4.2. LetΘbeanabstractobjectsystem. IfO ∼B Θ, lationbetweenΘandOcanbewellcharacterizedbyanatural thenOislinearizable. equivalencerelation,whichisaninstanceofbranchingbisimu- lation[17]forconcurrentobjects. Branchingbisimulationisan Proof: Because O ∼B Θ, from Definition 4.1, it is easy to action-basedversionofstutterbisimulation[18,19]. see ∼B preserves trace equivalence, i.e., trace(O) = trace(Θ). Therefore,O(cid:118) Θ. ByTheorem3.3,Oislinearizable. (cid:117)(cid:116) tr Definition4.1. Let∆ = (S ,−→,Act,ς)(i = 1,2)beobject i i i i i Theresultshowsthatlinearizabilitycanbeprovenifwecan systems(abstractorconcrete). AsymmetricrelationR ⊆ S1× establish the branching bisimulation equivalence ∼B between S isabranchingbisimulationforconcurrentobject,ifforany 2 anabstractobjectandaconcreteobject. (s ,s )∈R,thefollowingholds: 1 2 1. if s1−−→a s(cid:48)1, then there exists s(cid:48)2 such that s2−−→a s(cid:48)2 and 5. ProgressProperties (s(cid:48),s(cid:48))∈R, where a is a visible action (t,call,o.m(n)) or 1 2 (t,ret(n(cid:48)),o.m); 5.1. Divergence-sensitivebranchingbisimulation 2. if s1−−→τ s(cid:48)1,then(i)either(s(cid:48)1,s2)∈R,(ii)orthereexistsa Divergencemeanstheinfiniteτ-pathofasystem.Ifanobject finitepathfragment s2l1 ...lks(cid:48)2 (k≥0)suchthat s2==⇒τ lk ssiyssttsemofisntcultutedreτs-asnteipnsfi,ntihteeninttheernpaaltphawthi,lli.set.a,ypaftohrethvaetroinnlaylcooonp- −−→τ s(cid:48) and(s ,l)∈R(1≤i≤k)and(s(cid:48),s(cid:48))∈R; 2 1 i 1 2 withoutperforminganyobservableactionsuchas(t,ret(n),m). 4 We call the stutter path divergent. To distinguish infinite se- full transition relations from r , s and u ; and other transi- 3 3 3 riesofinternaltransitionsfromfiniteones,wetreatdivergence- tionswhicharenotrelevanttoourdiscussionareomitted. For sensitivebranchingbisimulation[14,17]. convenience, we write ”i” to denote τ, and (t ,call,dec) and 1 (t ,ret,dec) by dec and ret respectively (it is the same to 1 1 1 Definition5.1. Let ∆i = (Si,−→i,Act,ςi) be object systems inc2). Initially, the counter c = 0. Figure 3 (a) shows the ex- and R ⊆ S1 ×S2 an equivalence relation (such as ∼B) on the ecutions of ∆1, where each call by threads t1 and t2 can finish statesof∆1and∆2. theexecutioninafinitenumberofsteps. In(b), theexecution of dec is a self-loop and never returns. In (c), the execution • AstateuisR-divergentifthereexistsaninfinitepathfrag- 1 of dec is dependent, where at u and u it is a selp-loop that mentuu u ...suchthat(u,u)∈Rforalli>0. 1 1 3 1 2 i makesnoprogress,butafteru thattheatomicoperationτ up- 4 2 • Risdivergence-sensitiveifforany(u,v)∈R:ifuisdiver- datesc=1,t1evetuallyreturns. gentifandonlyifvisdivergent. In Figure 3 (b), the self-loop of ∆2 does not generate the return action ret , whereas ∆ in (a) can always do the re- 1 1 Definition5.2. If two states s and s in an object system ∆ turnactionret . Therefore, bycondition(1)ofDefinition4.1, 1 2 1 isdivergence-sensitivebranchingbisimilar,denotedby s ∼div r (cid:28) s . Thatis, ∼ candistinguishthedivergentstate s in 1 B 3 B 3 B 3 s ,ifthereexistsadivergence-sensitivebranchingbisimulation (b)andnon-divergentstater in(a). 2 3 relationRon∆suchthat(s ,s )∈R. (cid:117)(cid:116) However,inFigure3(c),whichisadependentprogresscon- 1 2 dition, it is easy to see r ∼ u , so r ∼ u . Thus, for this 7 B 5 3 B 3 Two systems ∆1 and ∆2 are divergence-sensitive branching case,∼B isnotsufficienttodistinguishtheself-loopofstateu3 bisimilar,iftheirinitialstatesarerelatedby∼div inthedisjoint in(c)fromthestater in(a).Howeverthenotionofdivergence- B 3 unionof∆1and∆2. sensitivebranchingbisimulation∼divcandistinguishanydiver- B gentstatefromnon-divergentstate,thatis,u (cid:28)div r . 3 B 3 5.2. Discussionsondivergenceofobjectsystems For the refinement relation (cid:118) , we found that it cannot dis- tr Thenotion∼div isavariantof∼ ,onlydifferinginthetreat- tinguish any divergent states, even when the simplest case in B B mentofdivergence. Wediscusstheabilityofthesenotions∼ , Figure 3 (a) and (b), where trace(s ) (cid:118) trace(r ). Therefore B 0 tr 0 ∼divand(cid:118) onidentifyingthedivergenceofobjectsystems. relation(cid:118) isnotsuitableforcheckinganyprogressproperties. B tr tr ConsiderthefollowingthreecountersinFigure2associated with methods inc and dec, where inc are the same, whereas 5.3. Checkingprogresspropertiesviabisimulation decareimplementedwithdistinctprogressconditions. There are various progress properties of concurrent objects, Counter∆ : suchaslock-freedomandwait-freedomfornon-blockingcon- 1 currency, and starvation-freedom and deadlock-freedom for int inc() int dec() blocking concurrency. These progress properties can all be atomic {c := c+1;} atomic {c := c-1;} return; return; specifiedinLTL(cid:13)(LTLwithoutnext)[15,24]. BecausethelinearizablespecificationΘshouldallowallpos- Counter∆ : 2 sibleinterleavingsandbeabletoterminatetheexecutionofits int inc() int dec() atomicblockatanystate,therealwaysexistssomethreadthat atomic {c := c+1;} while (1) do {skip;} canmakeprogressandperformthereturnactioninanyexecu- return; return; tion of Θ. This implies that any execution of abstract object Counter∆ : systemΘislock-free,i.e.,anyabstractobjectsystemΘislock- 3 free.Therefore,obtainingalock-freeabstractmodelisstraight- int dec() int inc() forwarddirectlyfromtheconcurrentspecification. while (1) do { atomic {c := c+1;} if (c>0) then c:=c-1;} return; return; Lemma5.3. ThelinearizablespecificationΘislock-free. Figure2: Differentprogresspropertiesofmethodsdec(). Proof: Θconsistsofasingleatomicblock(seeSection3.2),of whichtheinternalexecutioncorrespondstothecomputationof thesequentialspecificationthatbyassumptionisalwaystermi- Thedecofobject∆ iswait-free(andalsolock-free)which 1 nating. HenceforanyrunofΘ,therealwaysexistsonethread alwaysguaranteeseachcomputationmakesprogress. Thedec tocompleteitsmethodcallinfinitenumberofsteps. (cid:117)(cid:116) of ∆ does nothing and never makes progress. The dec of ∆ 2 3 is a dependent progress condition that makes progress only if To obtain wait-free abstract object systems, we need to en- c > 0. Thisprogressconditionof∆ isdependentonhowthe forcesomefairnessconstraintstothetransitionsystemstorule 3 systemschedulesthreads,whichisdifferentfromthelock-free outtheundesirablepaths. Thecommonfairproperties(suchas andwait-freepropertiesof∆ . strongfairness)canbeexpressedinnext-freeLTL[14]. 1 Assume that thread t invokes dec and t invokes inc once Itisknownthatdivergence-sensitivebranchingbisimulation 1 2 concurrently. The transition systems for ∆ , ∆ and ∆ are implies(next-free)LTL -equivalence[14]. Thisalsoholdsfor 1 2 3 (cid:13) partly presented in Figure 3 respectively, where we show the countablyinfinitetransitionsystemsthatarefinitelybranching. 5 u r 0 0 dec inc dec 1 inc 2 s 0 1 2 r 1 r 2 sdec 1 inc s 2 t 1 u 1 inc 2 u 2 inc t2 r 3 t t 1 1 inc 2 2 t u 3 t 2 1 2 s 1 ret 1 r 4 t 2 t 1 r 5 ret t 1 3 t 2 t 1 u 4 ret 2 2 s r 6 t 2 r 9ret 1 r 7 ret r210 t 1 r 8 t 1 4 ret s2 5 u ret 1 u 5 ret u2 t 1u 6 t 7 8 ret 2 r 11(a) Syret ste1m∆1. (b) Syste1m∆1. ret 2 u 9 (c)retSys1 tem∆1. Figure3: Divergence-sensitivityofbisimulationforobjectsystems. Thus, O ∼div Θ implies the preservation of all next-free LTL Memoryreclamation-hazardpointers. Wefurtherconsidera B formulas. Since the lock-freedom (and other progress proper- complicateconcurrentstackthatinvolveswithhazardpointers, ties [15, 24]) can be formulated in next-free LTL, for abstract whichprovidesamemoryreclamationmechanismtoavoidthe object Θ and concrete object O, the lock-free property can be ABA problem [27]. The hazard pointer variation is shown in preservedbytherelationO∼div Θ. Figure 4 (Lines 22-45), which includes push HP and pop HP. B Methodpush HPisthesameaspush.Methodpop HPinvolves Theorem5.4. Let O be a concrete object system, and Θ the a hazard pointer before the ABA-prone code (Line 37), and linearizablespecificationofO. callsthemethodRetire Node(o)(Line43)afterthesuccess- 1. IfO∼div Θ,thenOislinearizableandlock-free. ful CAS operation to determine which retired locations can be B 2. IfO ∼div ΘandΘiswait-free,thenOislinearizableand freed. The codes of Retire Node(o) can be found in [27]. B These operations are used to reclaim memory blocks safely, wait-free. whichisnotrelevanttotheeffectofthemethodcall. Furtherto Proof: Because O ∼div Θ, from Definitions 5.1 and 5.2, it is achievethedesiredlock-freedomofthestack,itisrequiredthat B easy to see ∼div preserves trace equivalence, i.e., trace(O) = methodRetire Node(o)shouldbewait-free. B trace(Θ). Therefore, O (cid:118) Θ. ByTheorem3.3, Oislineariz- tr able. Further, because divergence-sensitive branching bisimu- Figure5presentsanabstractstackasthespecification,where lation implies next-free LTL formula [14], the lock-free prop- ertycanbepreservedby∼div. ByLemma5.3,Θisalock-free eachmethodbodyofmethodspush specandpop specisim- B plementedbyanatomicblock. abstractobject,soifO∼div Θ,thenOislock-free. Thecaseof B wait-freepropertycanbeprovedsimilarly. (cid:117)(cid:116) Thelinearizationpointofpush/push HPandpop/pop HPof 6. ExperimentsonVerifyingLock-FreeStacks theconcurrentstacksisindependentofthedynamicexecution, whichcanbesimplylocatedontheimplementationcodesuch To show the efficiency and effectiveness of our method for thattheeffectofthemethodcallalwaystakesplaceatthestatic provinglinearizabilityandprogressproperties,weconductthe linearizationpointbetweenthemethodcallandthecorrespond- experimentonverifyingnon-blockingstacks[16,27,30]. We ingmethodreturn.Therefore,theeffectofeachmehtodcallcan employtheConstructionandAnalysisofDistributedProcesses beabstractedintotheatomicoperationofthelinearizablespec- (CADP)toolbox[20]tomodelandverifythealgorithms. ification;andviceversa,theatomicoperationofthespecifica- tion can be refined to several τ-steps including a linearization 6.1. Analyzingtheconcurrentstackwithhazardpointers pointoftheconcurrentstack. Suchthecorrespondingrelation Treiberstack. Figure4(Lines1-21)showstheTreiber’sstack can be naturally characterized by the branching bisimulation algorithm[16],whichinvolvestwomethodspushandpop.The equivalencedefinedinDefinition4.1. Ourexperimentconfirms stackisimplementedasalinked-listpointedbytheToppointer theexistenceofthebranchingbisimulationequivalencerelation (sharedvariable). Bothoperationsmodifythestackbydoinga betweentheconcurrentstack(withhazardpointers)inFigure4 CAS. The linearization point of push is Line 7 if the CAS suc- and theabstract stackin Figure5. Tocheck the progresscon- ceeds,andthelinearizationpointofpopiseitherLine14ifthe dition,theexplicitdivergenceinbranchingbisimulationisalso stackisempty;orLine18iftheCASsucceeds. needtobeconsidered. 6 class Node { int value; Node next; Node (int value) { value = value; next = null; } } Node Top; //Topisasharedvariable. void init() { Top := null } 32 pop_HP() { 33 done:=false; 01 void push(v) { 11 int pop() { 34 while(!done) { 22 void push_HP(v) { 02 bool done:=false; 12 bool done:=false; 35 Node old:=Top; 23 bool done:=false; 03 Node x:=new_node(v); 13 while(!done) { 36 if(old=null) return EMPTY; 24 Node x:=new_node(v); 04 while(!done) { 14 Node old:=Top; 37 *hp:=old; 25 while(!done) { 05 Node old:=Top; 15 if (old==null) 26 Node old:=Top; 38 if(Top==old) 06 x.next:=old; 16 then return EMPTY 27 x.next:=old; 39 x:=old.next; 07 done:=CAS(&Top,old,x); 17 Node x:=old.next; 40 done:=cas(&Top,old,x); 28 done:=CAS(&Top,old,x); 08 } 18 done:=CAS(&Top,old,x); 41 } 29 } 09 return; 19 } 42 v:=o.value; 30 return; 10 } 20 return old.data; 43 RetireNode(o); 31 } 21 } 44 return v; 45 } Figure4: Treiber’slock-freestackpush/popandthelock-freestackwithhazardpointerspush HP/pop HP,wheretheimplementa- tionofRetire Node(o)canbefoundin[27]. #th/#op ∆spec ∆Tr Thm5.4(linearizVabeirliifityc/altoicokn-ftirmeee)(insT)h(∆mT3r).3(linearizability) ∆HP Thm5.4(linearizVaberiliifityca/ltoiocnk-tfirmeee)(insT)h(∆mH3P.)3(linearizability) 2/1 70 81 0.73 0.80 195 0.76 0.97 2/2 487 823 0.74 0.92 7493 0.80 2.93 2/3 1678 3673 0.81 2.15 93352 1.45 25.30 2/4 4237 10999 0.93 5.08 808079 6.67 232.02 2/5 8920 26101 1.04 11.04 5447816 101.11 10514.94 2/6 16651 53197 1.20 21.89 31953747 283.87 >16h 2/7 28516 97435 1.63 39.55 174455921 1528.88 >10h 2/8 45769 164881 2.14 63.14 M.O. - - 3/1 706 1036 2.25 2.92 8988 2.51 9.11 3/2 12341 40309 1.09 21.48 4937828 76.26 >10h 3/3 77850 411772 4.11 230.60 M.O. - - 3/4 314392 2247817 19.90 >10h M.O. - - 4/1 6761 15595 0.71 11.36 612665 9.36 807.35 4/2 304197 2351919 22.52 4665.74 M.O. - - 5/1 64351 261527 10.48 816 60598453 408.27 >10h 6/1 616838 4771785 141.86 81316.58 M.O. - - Table1: Experimentalresultsonverifyingthelock-freestackswith/withouthazardpointers. bisimilar.Fortheautomatedverification,weconducttheexper- void push_spec(int x) atomic { imentonfinitesystems. Allexperimentsrunonaserverwhich Node node:=new Node(x); isequippedwitha4×[email protected] node.next:=Top; Top:=node; GBmemoryunder64-bitCentos7.2. } return; The verification results of both versions of the concurrent stacks are shown in Table 1, where M.O. means memory out. int pop_spec() The first column indicates the number of threads and opera- atomic { if (Top == null) b:=false tions. Thesecondandthirdcolumnsindicatethestatespaceof else thespecification∆ andTreiber’sstack∆ respectively;the Node Curr_Top:=Top; spec Tr Top:=Curr_Top.next; b:=true fourth and the fifth columns indicates the verification time (in } seconds)of∆ basedonbisimulationtechnique(Theorem5.4) if (b==false) Tr then return EMPTY andtracerefinementchecking(Theorem3.3). Thenextcolumn else return Curr_Top.data showsthestatespaceofthevariantstackwithhazardpointers ∆ .Andthelasttwocolumnsindicatetheverificationtime(in Figure5: Thelinearizablespecificationofconcurrentstacks. HP seconds)of∆ basedonthetwokindsofmethods. HP Fromtheexperimentalresults,wecanseethat,thebranching 6.2. Experimentalresults bisimulation method is much more feasible and efficient than tracerefinementcheckingforfinite-statesystems.Forexample, Toverifylinearizabilityandlock-freepropertyofconcurrent basedonthebisimulationequivalencechecking,theverification stacks,wecheckthattheabstractobjectinFigure5andthetwo of∆ withabout5millionstates(inthecase2/5)takesaround HP concreteobjectsinFigure4aredivergence-sensitivebranching 100 seconds, while trace refinement checking takes around 3 7 hours. Forthemostmillionstates,thetimeoftracerefinement Ourmethodisbuiltonthebehaviourofobjects,notontheir checkingisgreaterthan10hours. staticallysyntacticconstructs.Thus,foranyconcurrentobjects, wedonotcarehowtheobjectalgorithmisconstructed,butonly 6.3. Bugshunting-thestackwithrevisedhazardpointers[30] concernsstatetransitionrelationsbetweentheabstractandcon- Whileweverifythecorrectnessoftheconcurrentstackwith cretesystemmodels. Therefore,ourmethodcanbeappliedto therevisedhazardpointersin[30], acounter-example(atrace generalconcurrentobjectprograms. which witnesses the concrete object and the specification are Finally,weliketopointoutthatdivergence-sensitivebranch- notdivergence-sensitivebranchingbisimilar)isgivenbyCADP ing bisimulation implying LTL equivalence applies to any (cid:13) with just two concurrent threads. The revised version of the countabletransitionsystem.Thisimpliesthatallprogressprop- hazardpointersshowninFigure7avoidstheABAproblemat erties of abstract object programs are carried over to bisimilar theexpenseofviolatingthewait-freepropertyofhazardpoint- concreteones. Foranalysinginfinitesystems,branchingbisim- ersintheoriginalalgorithm[27]. Theerror-pathendsinaself- ulationcanbedoneusingstandardprooftechniquessuchaspa- loop (at state 20 in Figure 6) in which one thread keeps read- rameterisedbooleanequationsystems[26].Forfinite-statesys- ingthesamehazardpointervalueofanotherthreadinfunction tems,polynomial-timealgorithmscanbeemployed[25]. Note retireNode(t) without making any progress and causes the thatinthelattersetting,tracerefinementisPSPACE-complete. divergent path of the stack. So the revised stack in [30] is not lock-free. 7. RelatedWorkandConclusion Aswementionedintheintroduction,almostalltheworkin the literature on the verification of the correctness of concur- rentobjectsarebasedontherefinementtechniques. Ourwork takesthefirststeptoexplorethedivergence-sensitivebisimula- tionequivalencetechniquetoverifylinearizabilityandprogress propertiesofconcurrentobjects1. Ourtransitionsystemmodelsarecloselyrelatedto[8]. The keyideaof[8]istoconstructalinearizablespecificationusing labeledtransitionsystems(LTSs)anddescribelinearizabilityas the trace refinement relation between the specification and the concurrent algorithm. They also use abstract objects and state transition systems as semantic models for describing the be- haviourofconcurrentobjects,whichcorrespondtoourabstract Figure6: Thecounter-exampleoftracegeneratedbyCADP. andconcreteobjectsystems. However,thetracerefinementre- lationtheyproposedisonlysuitableforcheckinglinearizabil- ity, not for checking other properties e.g., progress properties. pop() { Moreover,theydonotdiscussthe(bi)-simulationrelationsbe- local done,next,t,t1; done:=false; tweenanabstractprogramandaconcreteobjectprogram. while(!done) { retireNode(t); Inourwork,werevealthatthereexistsanatural(branching) t:=Top; local i,t’; if(t==null) i:=1; bisimulationequivalencebetweentheabstractandtheconcrete return EMPTY; while(i<=th_num) { objectifweuseacoarse-grainedconcurrentobjectasthespec- HP[tid]:=t; if(i!=tid) { t1:=Top; t’:=HP[i]; ification. Moreover, when do model checking, for finite-state if(t==t1) { if (t’!=t) { systems,polynomial-timebranchingbisimulationcheckingal- next:=t.Next; i:=i+1;} done:=cas(&Top,t,next) } } gorithms is provided [25] - in contrast to (classical) PSPACE- } else i:=1+1; completetracerefinement-forcheckinglinearizability. HrPe[titirdeN]o:d=en(ultl);; }} Differentfrom[8],wherethetracerefinementisshowntobe return t; equivalentto linearizability, ourwork (Theorem4.2, Theorem } 5.4) shows that (divergence-sensitive) branching bisimulation Figure7: Abuggyimplementationofthelock-freestack[30]. implieslinearizability. Ifanobjectsystemandthespecification are(divergence-sensitive)branchingbisimilar,thentheobjectis linearizable;butiftheyarenot(divergence-sensitive)branching Summary. Our experiment presents the following advantages: bisimilar, thenitwillgenerateacounterexample, whichneeds (i)Weprovideauniformnotion∼dBivtoverifyboththelineariz- tobeanalyzedtodeterminewhetheritviolatesthelinearizabil- ability and lock-freedom of the variant of concurrent stacks; ity(e.g.,itmayviolatethelock-freeproperty). (ii)Ourtechniquesarefullyautomated(forfinite-statesystems) Other model checking methods on verifying linearizability and rely on the efficient existing branching bisimulation algo- are[21,22,23]. Cernyetal. [21]proposeanmethodautomata rithms. (iii) In contrast to proof techniques [3, 5, 28] for lin- earizabilty and progress, our method is able to generate coun- terexamplesinanautomatedmanner. 1Thispaperisarevisedversionofourtechnicalreport[9]. 8 to verify linearizability of concurrent linked-list implementa- [12] M.Herlihy,N.Shavit.TheArtofMultiprocessorProgramming.Morgan tions,butonlyfortheexecutionoftwofixedoperations.Vechev Kaufmann,April,2008. [13] D.Hendler,N.Shavit,L.Yerushalmi.AScalableLock-FreeStackAlgo- etal.[22]useSPINmodelcheckertomodelandverifylineariz- rithm.In:SPAA,pages206-215.2004. ability. Burckhardtetal. [23]presentanautomaticlinearizabil- [14] C.BaierandJ.-P.Katoen.PrinciplesofModelChecking.TheMITPress. itycheckerLine-UpbasedonthemodelcheckerCHESS. 2008. To verify progress properties, Gotsman etal. [7] first pro- [15] E.Petrank,M.Musuvathi,etal.ProgressGuaranteeforParallelPrograms viaBoundedLock-Freedom.In:PLDI,pages144-154.2009. pose a generalisation of linearizability such that it can specify [16] R.K.Treiber.SystemsProgramming:CopingwithParallelism.Technical liveness. Basedonthedefinition, theyrevealaconnectionbe- ReportRJ5118,IBMAlmadenResearchCenter,April1986. tweenlock-freepropertyandatermination-sensitivecontextual [17] RobJ.vanGlabbeekandW.PeterWeijland.BranchingTimeandAb- refinement, but they do not discuss other progress properties. stractioninBisimulationSemantics.JournaloftheACM.Vol.43(3):555- 600.1996. Liangetal. [6,28]studyfiveprogresspropertiesandshowthat [18] M. C. Browne, E. M. Clarke and O. Grumberg. Characterizing Finite each progress property is equivalent to a specific termination- Kripke Structures in Propositional Temporal Logic. Theoretical Com- sensitive contextual refinement. But they need to define dif- puterScience.Vol.59(1-2):115-131.1988. ferentrefinementrelationsfordifferentprogressproperties. In [19] KedarS.Namjoshi.ASimpleCharacterizationofStutteringBisimula- tion.In:FSTTCS,LNCS1346,pages284-296.1997. ourwork, divergence-sensitivebranchingbisimulationimplies [20] RaduMateescuandWendelinSerwe.ModelCheckingandPerformance LTL(cid:13)equivalenceandcanpreserveallprogresspropertiesthat EvaluationwithCADPIllustratedonShared-MemoryMutualExclusion are specified in LTL . So there is no need to define different Protocols.Vol.78(7):843-861.ScienceofComputerProgramming.2013. (cid:13) relations for different progress properties. Further, our recent [21] R.Colvin,L.Groves,V.LuchangcoandM.Moir.FormalVerificationofa LazyConcurrentList-BasedSetAlgorithm.In:CAV,LNCS4144,pages work[29]extendstheproposedbisimulationmethodsuchthat 475-488.2006. itcanverifymorecomplicatedconcurrentobjectsasin[5,28]. [22] M.T.Vechev,E.Yahav,andG.Yorsh.Experiencewithmodelchecking linearizability.In:SPIN,pages:261-278.2009. Conclusion. Thispaperproposesanovelandefficientmethod [23] S.Burckhardt,etal.Line-Up:ACompleteandAutomaticLinearizability based on divergence-sensitive branching bisimulation for ver- Checker.In:PLDI,pages330-340.2010. [24] B.Dongol.FormalisingProgressPropertiesofNon-BlockingPrograms. ifying linearizability and lock-free property of concurrent ob- In:ICFEM,pages284-303.2006. jects. Instead of using trace refinement to check linearizabil- [25] J.F.Groote,F.Vaandrager.AnEfficientAlgorithmforBranchingBisim- ity, our approach enables the use of branching bisimulation ulationandStutterEquivalence.In: Automata,LanguageandProgram- techniques for checking both the linearizability and progress ming,LNCS443,pages626-638.1990. [26] T.Chen,B.Ploeger,J.vandePolandT.Willemse.EquivalenceChecking propertiesofconcurrentobjects. Ourexperimentpresentsthat forInfiniteSystemsUsingParameterizedBooleanEquationSystems.In: (divergence-sensitive) branching bisimulation is an inherent CONCUR,vol.4703,pages:120-135.2007. equivalencerelationbetweentheTreiber’slock-freestack(and [27] M.M.Michael.HazardPointers: SafeMemoryReclamationforLock- FreeObjects.IEEETrans.ParallelDistrib.Syst.,15(6):491–504,2004. the revised stack with hazard pointers) and the corresponding [28] H.Liang,X.Feng,Z.Shao.Compositionalverificationoftermination- concurrent specification. Our method also reveals a new bug preservingrefinementofconcurrentprograms.CSL-LICS,2014: 65:1- violatingthelock-freedom. 65:10. [29] X.Yang, J.-P.Katoen, H.LinandH.Wu.ProvingLinearizabilityvia BranchingBisimulation.CoRRabs/1609.07546.Sep.,2016. References [30] M. Fu, Y. Li, X. Feng, Z. Shao, and Y. Zhang. Reasoning about Op- timisticConcurrencyUsingaProgramLogicforHistory.InCONCUR [1] M. Herlihy and J. Wing. Linearizability: A Correctness Condition for 2010,LNCSvol.6269,pages388–402.Springer,2010. Concurrent Objects. ACM Trans. on Program. Lang. and Syst. 12(3): 463-492.1990. [2] S. Owicki and L.Lamport. Proving Liveness Properties of Concurrent Programs.ACMTransactionsonProgrammingLanguagesandSystems. Vol.4(3):455-495.1982. [3] VafeiadisVictor.ModularFine-GrainedConcurrencyVerification.Cam- bridgeUniversity,PhdThesis.2008. [4] H.Liang,X.Feng,M.Fu.ARely-Guarantee-basedSimulationforVeri- fyingConcurrentProgramTransformations.In:POPL:455-468.2012. [5] H. Liang, X. Feng. Modular Verification of Linearizability with Non- FixedLinearizationPoints.In:PLDI,pages459-470.2013. [6] H.Liang,J.Hoffmann,X.Feng,Z.Shao.CharacterizingProgressProp- ertiesofConcurrentObjectsviaContextualRefinements.In:CONCUR, LNCS8052,pages:227-241.2013. [7] Alexey Gotsman, Hongseok Yang. Liveness-Perseving Atomicity Ab- straction.In:ICALP,LNCS:6756:453-465.2011. [8] Y.Liu,W.Chen,etal.VerifyingLinearizabiityviaOptmizedRefinement Checking.IEEETrans.onSoft.Eng.39(7):1018-1039.2013. [9] X.YangandJ.-P.Katoen.ProvingLinearizabilityandProgressofCon- current Objects by Bisimulation. Technical Report ISCAS-SKLCS-14- 16.Sep.2014. [10] P.Cerny,A.Radhakrishna,etal.ModelCheckingofLinearizabilityof ConcurrentListImplementations.In:CAV,pages465-479.2010. [11] I.Filipovic,P.O’Hearn,N.Rinetzky,H.Yang.AbstractionforConcur- rentObjects.TheoreticalComputerScience.Vol.411(51-52):4379-4398. 2010. 9

See more

The list of books you might like

Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.