Advent of Go Microservices

148 Pages·2020·1.208 MB·English

by  Tit Petric

Advent of Go Microservices
Tit Petric
Thisbookisforsaleathttp://leanpub.com/go-microservices
Thisversionwaspublishedon2020-02-17
ThisisaLeanpubbook.LeanpubempowersauthorsandpublisherswiththeLeanPublishing
process.LeanPublishingistheactofpublishinganin-progressebookusinglightweighttoolsand
manyiterationstogetreaderfeedback,pivotuntilyouhavetherightbookandbuildtractiononce
youdo.
©2019-2020TitPetric
Contents
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Aboutme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Whoisthisbookfor? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
HowshouldIstudyit? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
LinuxandDocker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Go:IntroductiontoProtobuf:Messages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
ProtobufsandGo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
GeneratingProtobufGocode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Wrappingup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Go:IntroductiontoProtobuf:Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Protobufservicedefinitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Ourmicroservice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
TwitchRPCscaffolding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Go:IntroductiontoProtobuf:gRPC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
GeneratingthegRPCscaffolding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Comparingtheserverside . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Comparingtheclientside . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Make:DynamicMakefiletargets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
BuildingmultipletargetswithGo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
DynamicMakefiletargetforbuildingRPC/Protobuffiles . . . . . . . . . . . . . . . . . . . . 19
Buildingourservicesfromcmd/. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Bash:Poormanscodegeneration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Generatingourservicecmd/folders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Generatingtheclientforourservice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Serverimplementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Bash:EmbeddingfilesintoGo. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Planningdatabasemigrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Poormansfileembedding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
CONTENTS
Go:Scaffoldingdatabasemigrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
ThemigrationAPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
DroneCI:Testingdatabasemigrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
ConfiguringDroneCIfordatabaseintegrationtests . . . . . . . . . . . . . . . . . . . . . . . 42
Go:Databasefirststructgeneration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Databaseschemadatastructuresandhelpers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Queryinginformationschema . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Typeconversions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Movingforward . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Go:Generatingdatabaseschemadocumentation . . . . . . . . . . . . . . . . . . . . . . . . . . 56
UpdatingtheGogenerators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
Implementingamarkdownoutputrenderer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
Acaveatemptoraboutdatabasemigrations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
Go:DependencyinjectionwithWire . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
TheGoway . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Runtimevs.compiletime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Wireanddependencyproviders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
WireanddependencyInjector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
Docker:Buildingimageswithsecurityinmind. . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
ConfiguringMakefiletargets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
ThebasicDockerfileimage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Securityimplicationsofourdockerimage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Possibleimprovements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Go:implementingamicroservice. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Thedatabaseschema . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Improvingthemicroserviceenvironment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
ImplementingPush. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
Docker/Bash:Issuingrequestsagainstourmicroservice . . . . . . . . . . . . . . . . . . . . . . 87
Thedocker-composefile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
Ourfirstrequest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Go:Improvingourdatabasehandling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Datasourcename-DSN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Multipledatabaseconnections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
Updatingdatabaseconnections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Databaseconnectionretry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Migrationimprovements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
Re-testingthebundledmigrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
CONTENTS
Go:InstrumentingtheHTTPservicewithElasticAPM . . . . . . . . . . . . . . . . . . . . . . 105
Wrappingourexistinghandler. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
Loggingerrorstoo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
SettingupELK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
SettingupAPM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Configuringoursender . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
ReviewingELKdata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
Go:InstrumentingtheDatabaseclientwithElasticAPM . . . . . . . . . . . . . . . . . . . . . 114
ExtendingDBconnection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
Verifyingit’sworking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Go:Stresstestingourservice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
Settingupourstresstest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
Resolvingdetectederrors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
Removingcontextcancellation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Wrappingup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
Go:Backgroundjobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
Addingbackgroundjobstoourservice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
DowereallyneedInit? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
Thebackgroundjob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
Housekeeping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Verifyingeverythingworks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
Go:Optimizingrequestswithaqueue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
TheQueue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
TheProducer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
TheConsumer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
Benchmarking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
Notesandpitfalls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
Introduction
About me
I’m passionate about API development, good practices, performance optimizations and educating
peopletostriveforquality.Ihaveabouttwodecadesofexperiencewritingandoptimizingsoftware,
andoftensolverealproblemsinvariousprogramminglanguages,Gobeingmyfavorite.
Youmightknowsomeofmyworkfrom:
• AuthorofAPIFoundationsinGo¹,
• Authorof12FactorApplicationswithDockerandGo²,
• Blogauthoronscene-si.org³
Professionally I specialize in writing APIs in the social/media industry and for various content
management products. Due to the public exposure of such APIs, it’s performance characteristics
areofparamountimportance.I’masolidMySQLDBAwithexperienceonotherdatabasesaswell.
I have strong opinions about PHP. I’ve spent the better part of two decades writing PHP code. I
have love for Go, and thankfully that doesn’t require me to have a super strong opinion, because
it’s harder to find really bad code. Honestly, the worst you can do is not handle errors and I’ll be
judgingyou.Disagree?Tweet@TitPetric⁴andletmeknow.
I’vewrittenaprofessionaldedicatedAPIframeworkwhichdoublesasansoftwaredevelopmentkit
for the Slovenian national TV and Radio station website, RTV Slovenia. The architecture being put
inplacewiththisbookismeanttoreplaceandimplementthemostcriticalmicroserviceswith.
Who is this book for?
This book is for everyone who writes applications for a living. I will cover a wide area of subjects
thatarerelatedtoeffectivemicroservicedevelopment.
You’ll super love this book if you’re into CI and DevOps. I hope to show you a bit of the world
outsideofGo,withexamplesofeffectiveapproachesyoucanusetodevelopyourmicroservices.
¹https://leanpub.com/api-foundations
²https://leanpub.com/12fa-docker-golang
³https://scene-si.org
⁴https://twitter.com/titpetric
Introduction 2
It is the intent of the book to provide a daily exercise which will teach you things about Drone CI,
bash, Makefiles, Docker, and anything else which might help you put together the foundation for
yourmicroserviceorapp.
Inthebook,Iwillcoverthesesubjects:
1. Go:IntroductiontoProtobuf:Messages
2. Go:IntroductiontoProtobuf:Services
3. Go:IntroductiontoProtobuf:gRPC
4. Make:DynamicMakefiletargets
5. Bash:Poormanscodegeneration
6. Bash:EmbeddingfilesintoGo
7. Go:Scaffoldingdatabasemigrations
8. DroneCI:Testingdatabasemigrations
9. Go:Databasefirststructgeneration
10. Go:Generatingdatabaseschemadocumentation
11. Go:DependencyinjectionwithWire
12. Docker:Buildingimageswithsecurityinmind
13. Go:implementingamicroservice
14. Docker/Bash:Issuingrequestsagainstourmicroservice
15. Go:Improvingourdatabasehandling
16. Go:InstrumentingtheHTTPservicewithElasticAPM
17. Go:InstrumentingtheDatabaseclientwithElasticAPM
18. Go:Stresstestingourservice
19. Go:Backgroundjobs
20. Go:Optimizingrequestswithaqueue
The idea for the book is to publish content as an advent calendar. Starting December 1st and then
everydayuntilchristmas,anewchapterwillbeadded.Thetitleslistedabovearealreadydoneand
areinthequeueforpublishing.
Following the step by step microservice development guide, we’ll learn how to use proto files to
scaffold your microservice, as well as how to use SQL-first code generation for our required data
structures.We’llimplementaTwitchRPCAPIwiththeminimalpossibledevelopmenteffort,which
willbefocusedontheeaseofdevelopmentforthefinalservice.
When you finish with the exercises in this book, you will have a solid framework for microservice
developmentandyoucanjustkeepaddingnewmicroservicestothemix.
Introduction 3
How should I study it?
Just go through the book from the start to finish. If possible, try to do the exercises yourself not by
copy pasting but by actually writing the code and snippets in the book, tailored to how you would
layoutyourproject.
The work in individual chapters builds on what was done in the previous chapter. The examples
are part of a step by step, chapter by chapter process, where we are developing the layout of the
microservice,andimplementingtechnicalrequirementsforallfuturemicroservices.
BesuretofollowtheRequirementssectionasyou’reworkingwiththebook.
Requirements
This is a book which gives you hands on instructions and examples of how to build a microservice
repository. We will be using a modern stack of software that will be required in order to complete
alltheexercises.
Linux and Docker
Theexamplesofthebookrelyonarecentdocker-engineinstallationonaLinuxhost.Linux,whilea
softrequirement,isreferencedmanytimesinthebookwithexamplesonhowtodownloadsoftware,
orhowwearebuildingourservices.YoucoulddothisonaMac,butyoumighthavetoadjustsome
thingsasyoumovethroughthechapters.
• Docker(arecentversion,I’dsuspectanythingafter18.09isfine),
• DockerCompose(arecentversionfromtheirGitHubpage),
• DroneCI(instructionstoinstalllaterinthebook),
• Variousshellutilitiesandprograms(awk,bash,sed,find,ls,make,…)
Pleaserefertotheofficialdockerinstallationinstructions⁵onhowtoinstallarecentdockerversion,
orinstallitfromyourpackagemanager.
We will use Docker Compose to pull in images for databases and whatever we need. As we have a
Docker-firstworkflow,thismeanswedon’tneedtodealwithpre-installingsoftware.
Own hardware
Therecommendedconfigurationifyouhaveyourownhardwareis:
• 2CPUcore,
• 2GBram,
• 128GBdisk(SSD)
Theminimalconfigurationknowntomostlyworkisabouthalfthat,butyoumightfindyourselfin
atightplaceassoonasyourusagegoesup.Ifyou’rejusttyingoutdocker,asimplevirtualmachine
mightbegoodenoughforyou,ifyou’renotrunningLinuxonyourlaptopalready.
⁵https://docs.docker.com/engine/installation/linux/
Requirements 5
Cloud quick-start
Ifhavingyourownhardwareisabitofabuzzkill,welcometotheworldofthecloud.Youcanliterally
set up your own virtual server on Digital Ocean within minutes. You can use this DigitalOcean
referrallink⁶togeta$10credit,whilealsohelpingmetakesomezerosofmyhostingbills.
After signing up, creating a Linux instance with a running Docker engine is simple, and only takes
a few clicks. There’s this nice green button on the top header of the page, where it says “Create
Droplet”.Clickit,andonthepageitopens,navigateto“One-clickapps”whereyouchoosea“Docker”
fromthelist.
ChooseDockerfrom“One-clickapps”
Running docker can be disk-usage intensive. Some docker images may “weigh” up to or more than
1 GB. I would definitely advise choosing an instance with at least 30GB of disk space, which is a
bargainfor$10amonth,butyouwillhavetokeepaneyeoutfordiskusage.It’sbeenknowntofill
up.
Chooseareasonabledisksize
Aside for some additional options on the page, like chosing a region where your droplet will be
running in, there’s only a big green “Create” button on the bottom of the page, which will set up
everythingyouneed.
⁶https://m.do.co/c/021b61109d56