The DevOps 2.0 Toolkit: Automating the Continuous Deployment Pipeline with Containerized Microservices

404 Pages·2016·8.1 MB·English

by  Viktor Farcic

The DevOps 2.0 Toolkit
Automating the Continuous Deployment Pipeline with
Containerized Microservices
Viktor Farcic
© 2016 Viktor Farcic
Tweet This Book!
PleasehelpViktorFarcicbyspreadingthewordaboutthisbookonTwitter!
Thesuggestedtweetforthisbookis:
IjustboughtTheDevOps2.0Toolkitby@vfarcic
Thesuggestedhashtagforthisbookis#devops2book.
Findoutwhatotherpeoplearesayingaboutthebookbyclickingonthislinktosearchforthis
hashtagonTwitter:
https://twitter.com/search?q=#devops2book
Contents
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
AbouttheAuthor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
TheDevOpsIdeal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
ContinuousIntegration,Delivery,andDeployment . . . . . . . . . . . . . . . . . . . . . 6
TheLightattheEndoftheDeploymentPipeline . . . . . . . . . . . . . . . . . . . . . . 9
The Implementation Breakthrough: Continuous Deployment, Microservices, and Con-
tainers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
ContinuousIntegration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
ContinuousDeliveryandDeployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Microservices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
TheThreeMusketeers:SynergyofContinuousDeployment,Microservices,andContainers 25
SystemArchitecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
MonolithicApplications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
ServicesSplitHorizontally . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Microservices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
MonolithicApplicationsandMicroservicesCompared . . . . . . . . . . . . . . . . . . . 33
DeploymentStrategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
MicroservicesBestPractices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
FinalThoughts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
SettingUptheDevelopmentEnvironmentWithVagrantandDocker . . . . . . . . . . . 51
CombiningMicroserviceArchitectureandContainerTechnology . . . . . . . . . . . . . 51
VagrantandDocker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
DevelopmentEnvironmentSetup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
ImplementationoftheDeploymentPipeline:InitialStages . . . . . . . . . . . . . . . . . . 63
SpinningUptheContinuousDeploymentVirtualMachine . . . . . . . . . . . . . . . . . 63
CONTENTS
DeploymentPipelineSteps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
TheChecklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
ConfigurationManagementintheDockerWorld . . . . . . . . . . . . . . . . . . . . . . . 79
CFEngine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Puppet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Chef . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
Ansible . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
FinalThoughts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
ConfiguringtheProductionEnvironment . . . . . . . . . . . . . . . . . . . . . . . . . . 82
SettingUptheAnsiblePlaybook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
ImplementationoftheDeploymentPipeline:IntermediateStages . . . . . . . . . . . . . . 89
DeployingContainerstotheProductionServer . . . . . . . . . . . . . . . . . . . . . . . 90
DockerUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
TheChecklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
ServiceDiscovery:TheKeytoDistributedServices . . . . . . . . . . . . . . . . . . . . . . 98
ServiceRegistry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
ServiceRegistration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
ServiceDiscovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
ServiceDiscoveryTools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
ManualConfiguration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
Zookeeper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
etcd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
Consul . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
ServiceDiscoveryToolsCompared . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
ProxyServices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
ReverseProxyService . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
HowCanProxyServiceHelpOurProject? . . . . . . . . . . . . . . . . . . . . . . . . . . 146
nginx . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
HAProxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
ProxyToolsCompared . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
ImplementationoftheDeploymentPipeline:TheLateStages . . . . . . . . . . . . . . . . 168
StartingtheContainers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
IntegratingtheService . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
RunningPost-DeploymentTests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
PushingtheTestsContainertotheRegistry . . . . . . . . . . . . . . . . . . . . . . . . . 174
TheChecklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
AutomatingImplementationoftheDeploymentPipeline . . . . . . . . . . . . . . . . . . 177
DeploymentPipelineSteps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
CONTENTS
ThePlaybookandtheRole . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
Pre-DeploymentTasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
DeploymentTasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
Post-DeploymentTasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
RunningtheAutomatedDeploymentPipeline . . . . . . . . . . . . . . . . . . . . . . . . 189
ContinuousIntegration(CI),DeliveryandDeployment(CD)Tools . . . . . . . . . . . . . 191
CI/CDToolsCompared . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
TheShortHistoryofCI/CDTools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Jenkins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
FinalThoughts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
Blue-GreenDeployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
TheBlue-GreenDeploymentProcess . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
ManuallyRunningtheBlue-GreenDeployment . . . . . . . . . . . . . . . . . . . . . . . 226
AutomatingtheBlue-GreenDeploymentwithJenkinsWorkflow . . . . . . . . . . . . . 243
ClusteringAndScalingServices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
AxisScaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
DockerClusteringToolsCompared:KubernetesvsDockerSwarmvsMesos . . . . . . . 259
DockerSwarmWalkthrough . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
DeployingwithDockerSwarmWithoutLinks . . . . . . . . . . . . . . . . . . . . . . . . 278
ScalingServiceswithDockerSwarm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
SchedulingContainersDependingonReservedCPUsandMemory . . . . . . . . . . . . 287
AutomatingDeploymentWithDockerSwarmandAnsible . . . . . . . . . . . . . . . . . 292
Self-HealingSystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
Self-HealingLevelsandTypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
Self-HealingArchitecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
Self-HealingwithDocker,ConsulWatches,andJenkins. . . . . . . . . . . . . . . . . . . 313
CombiningOn-PremisewithCloudNodes . . . . . . . . . . . . . . . . . . . . . . . . . . 346
Self-HealingSummary(SoFar) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346
CentralizedLoggingandMonitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348
TheNeedforCentralizedLogging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
SendingLogEntriestoElasticSearch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351
ParsingLogEntries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
SendingLogEntriestoaCentralLogStashInstance . . . . . . . . . . . . . . . . . . . . . 362
SendingDockerLogEntriestoaCentralLogStashInstance . . . . . . . . . . . . . . . . . 368
Self-HealingBasedonSoftwareData . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
LoggingHardwareStatus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387
Self-HealingBasedonHardwareData . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393
CONTENTS
FinalThoughts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394
Farewell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396
Preface
I started my career as a developer. During those early days, all I knew (and thought I should
know) was to write code. I thought that a great software designer is a person that is proficient
inwritingcodeandthatthepathtothemasteryofthecraftwastoknoweverythingaboutasingle
programminglanguageofchoice.Lateron,thatchangedandIstartedtakinganinterestindifferent
programming languages. I switched from Pascal to Basic and then ASP. When Java and, later on,
.Net came into existence, I learned benefits of object oriented programming. Python, Perl, Bash,
HTML, JavaScript, Scala… Each programming language brought something new and thought me
howtothinkdifferentlyandhowtopicktherighttoolforthetaskathand.Witheachnewlanguage
Ilearned,IfeltlikeIwasclosertobeinganexpert.AllIwantedwastobecomeaseniorprogrammer.
Thatdesirechangedwithtime.IlearnedthatifIwastodomyjobwell,Ihadtobecomeasoftware
craftsman.Ihadtolearnmuchmorethantotypecode.Testingbecamemyobsessionforsometime,
andnowIconsideritanintegralpartofdevelopment.Exceptinveryspecialcases,eachlineofcodeI
writeisdonewithtest-drivendevelopment(TDD).Itbecameanindispensablepartofmytool-belt.I
alsolearnedthatIhadtobeclosetothecustomerandworkwithhimsidebysidewhiledefiningwhat
shouldbedone.Allthatandmanyotherthingsleadmetosoftwarearchitecture.Understandingthe
“big picture” and trying to fit different pieces into one big system was the challenge that I learned
tolike.
Throughout all the years I’ve been working in the software industry, there was no single tool,
frameworkorpracticethatIadmiredmorethancontinuousintegration(CI)and,lateron,continuous
delivery (CD). The real meaning of that statement hides behind the scope of what CI/CD envelops.
In the beginning, I thought that CI/CD means that I knew Jenkins and was able to write scripts.
As the time passed and I got more and more involved and learned that CI/CD relates to almost
every aspect of software development. That knowledge came at a cost. I failed (more than once)
to create a successful CI pipeline with applications I worked with at the time. Even though others
considered the result a success, now I know that it was a failure because the approach I took was
wrong.CI/CDcannotbedonewithoutmakingarchitecturaldecisions.Similarcanbesaidfortests,
configurations,environments,fail-over,andsoon.TocreateasuccessfulimplementationofCI/CD,
weneedtomakealotofchangesthat,onthefirstlook,donotseemtobedirectlyrelated.Weneed
toapplysomepatternsandpracticesfromtheverybeginning.Wehavetothinkaboutarchitecture,
testing,coupling,packaging,faulttolerance,andmanyotherthings.CI/CDrequiresustoinfluence
almosteveryaspectofsoftwaredevelopment.Thatdiversityiswhatmademefallinlovewithit.By
practicingCI/CDweareinfluencingandimprovingalmosteveryaspectofthesoftwaredevelopment
lifecycle.
To be truly proficient with CI/CD, we need to be much more than experts in operations. DevOps
movementwasasignificantimprovementthatcombinedtraditionaloperationswithadvantagesthat
development could bring. I think that is not enough. We need to know and influence architecture,
1
Preface 2
testing,development,operationsandevencustomernegotiationsifwewanttogainallthebenefits
that CI/CD can bring. Even the name DevOps as the driving force behind the CI/CD is not
suitable since it’s not only about development and operations but everything related to software
development. It should also include architects, testers, and even managers. DevOps was a vast
improvement when compared to the traditional operations by combining them with development.
Themovementunderstoodthatmanuallyrunningoperationsisnotanoptiongivencurrentbusiness
demandsandthatthereisnoautomationwithoutdevelopment.Ithinkthatthetimecametoredefine
DevOps by extending its scope. Since the name DevOpsArchTestManageAndEverythingElse is too
cumbersometorememberandclosetoimpossibletopronounce,IoptforDevOps2.0.It’sthenext
generationthatshoulddroptheheavydo-it-allproductsforsmallertoolsdesignedtodoveryspecific
tasks.It’stheswitchthatshouldgobacktothebeginningandnotonlymakesurethatoperationsare
automatedbutthatthewholesystemisdesignedinthewaythatitcanbeautomated,fast,scalable,
fault-tolerant,withzero-downtime,easytomonitor,andsoon.Wecannotaccomplishthisbysimply
automating manual procedures and employing a single do-it-all tool. We need to go much deeper
thanthatandstartrefactoringthewholesystembothonthetechnologicalaswellastheprocedural
level.
Overview
Thisbookisaboutdifferenttechniquesthathelpusarchitectsoftwareinabetterandmoreefficient
waywithmicroservicespackedasimmutablecontainers,testedanddeployedcontinuouslytoservers
that are automatically provisioned with configuration management tools. It’s about fast, reliable
and continuous deployments with zero-downtime and ability to roll-back. It’s about scaling to any
number of servers, design of self-healing systems capable of recuperation from both hardware and
softwarefailuresandaboutcentralizedloggingandmonitoring ofthecluster.
In other words, this book envelops the whole microservices development and deployment lifecycle
using some of the latest and greatest practices and tools. We’ll use Docker, Kubernetes, Ansible,
Ubuntu, Docker Swarm and Docker Compose, Consul, etcd, Registrator, confd, Jenkins, and so on.
We’llgothroughmanypracticesand,evenmore,tools.
Finally,whiletherewillbealotoftheory,thisisahands-onbook.Youwon’tbeabletocompleteit
byreadingitinametroonawaytowork.You’llhavetoreadthisbookwhileinfrontofacomputer,
and get your hands dirty. Eventually, you might get stuck and in need of help. Or you might want
towritearevieworcommentonthebook’scontent.PleasepostyourthoughtsontheTheDevOps
2.0ToolkitchannelinDisqus¹.Ifyoupreferone-on-onediscussion,feelfreetosendmeanemailto
[email protected],ortocontactmeonHangOuts,andI’llgivemybesttohelpyouout.
¹https://disqus.com/home/channel/thedevops20toolkit/
3