The Practice of Cloud System Administration: Devops and Sre Practices for Web Services, Volume 2

559 Pages·2014·5.605 MB·English

by  Limoncelli, Thomas A;Chalup, Strata R;Hogan, Christina J

The Practice of Cloud
System Administration
Volume 2
This page intentionally left blank
The Practice of Cloud
System Administration
DevOps and SRE Practices for
Web Services
Volume 2
Thomas A. Limoncelli
Strata R. Chalup
Christina J. Hogan
UpperSaddleRiver,NJ(cid:15)Boston(cid:15)Indianapolis(cid:15)SanFrancisco
NewYork(cid:15)Toronto(cid:15)Montreal(cid:15)London(cid:15)Munich(cid:15) Paris(cid:15)Madrid
Capetown(cid:15)Sydney(cid:15)Tokyo(cid:15)Singapore(cid:15)MexicoCity
Manyofthedesignationsusedbymanufacturersandsellerstodistinguishtheirproductsareclaimed
astrademarks.Wherethosedesignationsappearinthisbook,andthepublisherwasawareofatrade-
markclaim,thedesignationshavebeenprintedwithinitialcapitallettersorinallcapitals.
Theauthorsandpublisherhavetakencareinthepreparationofthisbook,butmakenoexpressed
orimpliedwarrantyofanykindandassumenoresponsibilityforerrorsoromissions.Noliabilityis
assumedforincidentalorconsequentialdamagesinconnectionwithorarisingoutoftheuseofthe
informationorprogramscontainedherein.
Forinformationaboutbuyingthistitleinbulkquantities,orforspecialsalesopportunities(whichmay
includeelectronicversions;customcoverdesigns;andcontentparticulartoyourbusiness,training
goals,marketingfocus,orbrandinginterests),pleasecontactourcorporatesalesdepartmentatcorp-
[email protected][phone].
Forgovernmentsalesinquiries,[email protected].
ForquestionsaboutsalesoutsidetheUnitedStates,[email protected].
VisitusontheWeb:informit.com/aw
LibraryofCongressCataloging-in-PublicationData
Limoncelli,Tom.
Thepracticeofcloudsystemadministration:designingandoperatinglargedistributedsystems/
ThomasA.Limoncelli,StrataR.Chalup,ChristinaJ.Hogan.
volumes cm
Includesbibliographicalreferencesandindex.
ISBN-13:978-0-321-94318-7(volume2:paperback)
ISBN-10:0-321-94318-X(volume2:paperback)
1.Computernetworks—Management.2.Computersystems.3.Cloudcomputing.4.Electronicdata
processing—Distributedprocessing.I.Chalup,StrataR.II.Hogan,ChristinaJ.III.Title.
TK5105.5.L5292015
004.67’82068—dc23 2014024033
Copyright©2015ThomasA.Limoncelli,Virtual.NETInc.,ChristinaJ.LearnéeHogan
Allrightsreserved.PrintedintheUnitedStatesofAmerica.Thispublicationisprotectedbycopyright,
andpermissionmustbeobtainedfromthepublisherpriortoanyprohibitedreproduction,storageina
retrievalsystem,ortransmissioninanyformorbyanymeans,electronic,mechanical,photocopying,
recording,orlikewise.Toobtainpermissiontousematerialfromthiswork,pleasesubmitawritten
requesttoPearsonEducation,Inc.,PermissionsDepartment,200OldTappanRoad,OldTappan,New
Jersey07675,oryoumayfaxyourrequestto(201)236-3290.
ISBN-13:978-0-321-94318-7
ISBN-10:0-321-94318-X
3 17
Contents at a Glance
Contents vii
Preface xxiii
AbouttheAuthors xxix
Introduction 1
Part I Design: Building It 7
Chapter1 DesigninginaDistributedWorld 9
Chapter2 DesigningforOperations 31
Chapter3 SelectingaServicePlatform 51
Chapter4 ApplicationArchitectures 69
Chapter5 DesignPatternsforScaling 95
Chapter6 DesignPatternsforResiliency 119
Part II Operations: Running It 145
Chapter7 OperationsinaDistributedWorld 147
Chapter8 DevOpsCulture 171
Chapter9 ServiceDelivery:TheBuildPhase 195
Chapter10 ServiceDelivery:TheDeploymentPhase 211
Chapter11 UpgradingLiveServices 225
Chapter12 Automation 243
Chapter13 DesignDocuments 275
Chapter14 Oncall 285
Chapter15 DisasterPreparedness 307
Chapter16 MonitoringFundamentals 331
v
vi ContentsataGlance
Chapter17 MonitoringArchitectureandPractice 345
Chapter18 CapacityPlanning 365
Chapter19 CreatingKPIs 387
Chapter20 OperationalExcellence 401
Epilogue 417
Part III Appendices 419
AppendixA Assessments 421
AppendixB TheOriginsandFutureofDistributedComputing
andClouds 451
AppendixC ScalingTerminologyandConcepts 475
AppendixD TemplatesandExamples 481
AppendixE RecommendedReading 487
Bibliography 491
Index 499
Contents
Preface xxiii
AbouttheAuthors xxix
Introduction 1
Part I Design: Building It 7
1 DesigninginaDistributedWorld 9
1.1 VisibilityatScale 10
1.2 TheImportanceofSimplicity 11
1.3 Composition 12
1.3.1 LoadBalancerwithMultipleBackendReplicas 12
1.3.2 ServerwithMultipleBackends 14
1.3.3 ServerTree 16
1.4 DistributedState 17
1.5 TheCAPPrinciple 21
1.5.1 Consistency 21
1.5.2 Availability 21
1.5.3 PartitionTolerance 22
1.6 LooselyCoupledSystems 24
1.7 Speed 26
1.8 Summary 29
Exercises 30
vii
viii Contents
2 DesigningforOperations 31
2.1 OperationalRequirements 31
2.1.1 Configuration 33
2.1.2 StartupandShutdown 34
2.1.3 QueueDraining 35
2.1.4 SoftwareUpgrades 36
2.1.5 BackupsandRestores 36
2.1.6 Redundancy 37
2.1.7 ReplicatedDatabases 37
2.1.8 HotSwaps 38
2.1.9 TogglesforIndividualFeatures 39
2.1.10 GracefulDegradation 39
2.1.11 AccessControlsandRateLimits 40
2.1.12 DataImportControls 41
2.1.13 Monitoring 42
2.1.14 Auditing 42
2.1.15 DebugInstrumentation 43
2.1.16 ExceptionCollection 43
2.1.17 DocumentationforOperations 44
2.2 ImplementingDesignforOperations 45
2.2.1 BuildFeaturesinfromtheBeginning 45
2.2.2 RequestFeaturesasTheyAreIdentified 46
2.2.3 WritetheFeaturesYourself 47
2.2.4 WorkwithaThird-PartyVendor 48
2.3 ImprovingtheModel 48
2.4 Summary 49
Exercises 50
3 SelectingaServicePlatform 51
3.1 LevelofServiceAbstraction 52
3.1.1 InfrastructureasaService 52
3.1.2 PlatformasaService 54
3.1.3 SoftwareasaService 55
3.2 TypeofMachine 56
3.2.1 PhysicalMachines 57
3.2.2 VirtualMachines 57
3.2.3 Containers 60
Contents ix
3.3 LevelofResourceSharing 62
3.3.1 Compliance 63
3.3.2 Privacy 63
3.3.3 Cost 63
3.3.4 Control 64
3.4 Colocation 65
3.5 SelectionStrategies 66
3.6 Summary 68
Exercises 68
4 ApplicationArchitectures 69
4.1 Single-MachineWebServer 70
4.2 Three-TierWebService 71
4.2.1 LoadBalancerTypes 72
4.2.2 LoadBalancingMethods 74
4.2.3 LoadBalancingwithSharedState 75
4.2.4 UserIdentity 76
4.2.5 Scaling 76
4.3 Four-TierWebService 77
4.3.1 Frontends 78
4.3.2 ApplicationServers 79
4.3.3 ConfigurationOptions 80
4.4 ReverseProxyService 80
4.5 Cloud-ScaleService 80
4.5.1 GlobalLoadBalancer 81
4.5.2 GlobalLoadBalancingMethods 82
4.5.3 GlobalLoadBalancingwithUser-SpecificData 82
4.5.4 InternalBackbone 83
4.6 MessageBusArchitectures 85
4.6.1 MessageBusDesigns 86
4.6.2 MessageBusReliability 87
4.6.3 Example1:Link-ShorteningSite 87
4.6.4 Example2:EmployeeHumanResourcesDataUpdates 89
4.7 Service-OrientedArchitecture 90
4.7.1 Flexibility 91
4.7.2 Support 91
4.7.3 BestPractices 91