Scaling PHP Applications

210 Pages·2014·5.646 MB·English

by  Steve Corona

Scaling PHP
Steve Corona
Thisbookisforsaleathttp://leanpub.com/scalingphp
Thisversionwaspublishedon2014-05-23
ThisisaLeanpubbook.LeanpubempowersauthorsandpublisherswiththeLeanPublishing
process.LeanPublishingistheactofpublishinganin-progressebookusinglightweighttools
andmanyiterationstogetreaderfeedback,pivotuntilyouhavetherightbookandbuild
tractiononceyoudo.
©2012-2014SteveCorona
Contents
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Howthisbookcametobe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Howthisbookisdesigned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Whoshouldreadthisbook? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Whatyouneedforthisbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Whatyou’lllearn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Whyyouneedthisbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
RuleNumeroUno . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
DitchingtheLAMPStack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
What’swrongwithLAMP? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
TheScalableStack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
DNS:Whyyouneedtocare . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
DNSLoadDistribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
DNSResolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
LoadBalancingwithHAProxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
HAProxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Choosingthebesthardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Automaticfailoverwithkeepalived . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Tuninglinuxforanetworkheavyload . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Issuesatscale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
AppServer:HorizontalScalingwithNginxandPHP-FPM . . . . . . . . . . . . . . . . 26
ChoosingtherightversionofPHP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
NginxandPHP-FPM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Choosingthebesthardwareforyourapplicationserver . . . . . . . . . . . . . . . . . 33
UsingaPHPOpcodecache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
TuningLinuxforPHP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Scalingsessionhandling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
DatabaseServer:UltimateMySQLTuningGuide . . . . . . . . . . . . . . . . . . . . . 46
GettingNoSQLPerformanceoutofMySQL . . . . . . . . . . . . . . . . . . . . . . . . 46
Dealingwithlibmysql’sinabilitytosettimeouts . . . . . . . . . . . . . . . . . . . . . 49
TuningyourMySQLConfiguration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
TuningLinuxforanintensivedatabase . . . . . . . . . . . . . . . . . . . . . . . . . . 63
LoadbalancingMySQLSlaves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
CONTENTS
AcceptingtheMasterasaSPOF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
UnderstandingissueswithNUMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Choosingthebesthardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
OnlineSchemaChanges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
FurtherTopics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
CacheServer:UsingRedis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
ChoosingbetweenRedis,Memcached,andAPC . . . . . . . . . . . . . . . . . . . . . 85
RedisCommands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
TheimportanceofAtomicOperations . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
PerformanceLimitationsofRedis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
InstallingRedisfromDotdeb . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
InstallingthephpredisCextension . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
TuningRedisforPerformance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
ScalingRedistoMultipleServers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
“TheDogpile” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
RussianDollCaching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
RedisBitmaps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
RedisNotificationFeeds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
WorkerServer:AsynchronousResqueWorkers . . . . . . . . . . . . . . . . . . . . . . 109
WhyuseQueues? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
GettingstartedwithResqueandphp-resque . . . . . . . . . . . . . . . . . . . . . . . 111
WritingaResqueJob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
AlternativestoResque . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
Thingsyoushouldputinaqueue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
CodingandDebuggingatScale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
Scalingyourcode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
Capistranoforcodedeployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
LiveDebuggingPHPwithstrace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
PartingAdvice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
Wanttoupgrade? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
SendingFeedbackandTestimonials . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
Intoodeep?NeedhelpASAP? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
Updates? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
AbouttheAuthor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
CaseStudy:Horizontallyscalinguseruploads . . . . . . . . . . . . . . . . . . . . . . . 133
CaseStudy:HowDNStookdownTwitpicdownfor4hours . . . . . . . . . . . . . . . 138
CaseStudy:ScalingMySQLVertically(10to3MySQLServersw/SSDs) . . . . . . . . 139
CaseStudy:HotMySQLBackupsw/Percona . . . . . . . . . . . . . . . . . . . . . . . 141
MovingfrommysqldumptoPerconaXtraBackup . . . . . . . . . . . . . . . . . . . . . 143
CaseStudy:AsyncAPIsw/Kestrelsavedmybutt . . . . . . . . . . . . . . . . . . . . . 145
CONTENTS
CaseStudy:ThemissingguidetoMemcached . . . . . . . . . . . . . . . . . . . . . . . 148
ChoosingMemcachedoverRedis? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
InstallingMemcached . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
HookingupPHPtoMemcached . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
CompressionandSerialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
ServerPools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
AtomicCounters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
AvoidingtheRatRace:CASTokens . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
Watchoutfortimeouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
UsingMemcachedforSessions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
DebuggingMemcachedfromTelnet . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
MyMemcachedSetup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
FurtherReadingandTools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
CaseStudy:HTTPCachingandtheNginxFastcgiCache . . . . . . . . . . . . . . . . . 166
CachingStaticAssets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
CachingProxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
CaseStudy:CakePHPFrameworkCaching . . . . . . . . . . . . . . . . . . . . . . . . . 173
CakePHP,youdevilishdessert . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
ThelifeanddeathofaCakePHPrequest . . . . . . . . . . . . . . . . . . . . . . . . . 174
Ifitsmellslikearacecondition… . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
Bustingtheassumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
Nowwhat?I’moutofideas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
Thesolution(s) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
CaseStudy:OptimizingimagehandlinginPHP . . . . . . . . . . . . . . . . . . . . . . 181
Thenaive“PHPTutorial”way . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
ImageMagickvsGraphicsMagick . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
AllowingLargeFileUploads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
StrippingEXIFMetadata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
Autorotation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
CaseStudy:BenchmarkingandLoadTestingWebServices . . . . . . . . . . . . . . . 196
SettingupaTestEnvironment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
Thefamousab(ApacheBenchmark) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
Siege,amoremodernab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
Beeswithmachineguns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
Sysbench . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
Sponsors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
Preface
How this book came to be
In2009,ImetNoahEverett,founderandCEOofTwitpicthroughaposthemadeonTwitter.Noah
is a smart, self-taught programmer, quite literally thrown into scaling when Twitpic launched
into overnight success after the first picture of the “Miracle on the Hudson”,US Airways Flight
1549¹waspostedonTwitpic².
Subsequently, our first conversation was less of an “interview” and more me introducing Noah
tomemcacheandhowitcouldbesetupwithinthenexthour—uptillthatpointthesitecrashed
regularly,andwasdownduringourphonecall.Iwasn’ttherewithhimatthetime,butIliketo
thinkhewassweatingbullets,feverishlytypingapt-get install memcachewhilehundredsof
angrybeiber-fansweresendingnegativeenergy(andtweets)hisway.Anhouroffreeconsulting
for a life changing opportunity to own the tech behind one of the top 100 sites on the internet?
Soundsgoodtome.ItwasagreattimetobeatTwitpic.
There’saplaneintheHudson.I’montheferrygoingtopickupthepeople.Crazy.-JanisKrums(@jkrums)
When I started at Twitpic, I quickly learned how big of a mess it was. The infrastructure was
a mix of different servers and FreeBSD versions, backed up by a steaming pile of PHP-inlined
HTML. Remember, Twitpic was built in a Red Bull fueled weekend, a side project for Noah
to share pictures with a couple of his friends. It wasn’t meant win the prettiest-code beauty
pageant. But, wow, it was bad. Think of the type of PHP your 14 year old little-brother might
write— spaghetti, pagename.php files, no MVC framework, and a handful of include_once’s at
thetopofeachfile.
¹http://en.wikipedia.org/wiki/US_Airways_Flight_1549
²http://twitpic.com/135xa
Preface 2
Inthebeginning,NoahandIhadtoscheduleourlivesaroundourlaptops.Goingtothegrocery
store?Bringthelaptop.Headedoutonadate?Don’tforgetthelaptop.
“Hey,Iknowthisdateisgoingawesome,butIneedtoputitonpauseandjumpon
mylaptopreallyquick.DoyouthinktheyhavefreeWiFihere?”
Twitpicwaslikeajealousgirlfriend,waitingtoruinyourdayinamomentsnotice.Ineverknew
whenI’dhavetopopintoaStarbuckstorestartApache.
Mostly out of necessity, and because no one enjoys cuddling with their laptop at night, we
learned.Wescaled.Weimprovedourinfrastructure.Bynomeansisoursetupperfect,butwe’ve
learned an incredible amount, and after three years I’ve built a pretty solid architecture. What
happens when servers go down in the middle of the night today? Nothing. I sleep peacefully
throughthenight,becausewe’vebuiltahorizontal,replicatedsystemthatisdesignedforfailure.
How this book is designed
Thisbook,althoughPHPcentric,canbeappliedtoanylanguage.Mostofthetricks,techniques,
and design patterns will apply cleanly to Ruby, Python, Java or really anything besides Visual
Basic(ok,maybeevenVBtoo).
Readitlikeacookbook,ormaybeevenachooseyourownadventurestory—youdon’tneedto
readitinorder,orcovertocover.Jumparoundtodifferentchaptersorsections,diggingintothe
ones that pique your interest the most. Or, if you’re like Noah, the one you need right now to
getyoursitebackonline.
I’velaidoutthechapterssotheymovefromtheoutsideofthestack,level-by-level,subsequently
getting deeper until we hit the code. Yea, the code is last. Why? Because it’s usually the fastest
part of your stack. I get it, you’re not a sysadmin and you don’t play one on TV either, but 80%
ofyourscalinggainsaretobehadoutsideofyourcodeandalittlebitofLinux-fucangoalong
way.
Who should read this book?
Thisbookisdesignedforstartups,entrepreneursandsmartpeoplethatlovetohustleandbuild
thingstherightway.
You should know PHP, your current stack and your way around Linux. The examples assume
UbuntuServer,butanylinuxdistributionwillgetthejobdone.Ifyoucanuseapackagemanage
(apt-get,yum)andedittextfiles,you’llbeabletoimplementmostexamplesinthisbook.
My readers are smart and intelligent people that don’t need their hands held. I explain
complicated topics and provide thorough examples, but this is NOT simply regurgitated
documentation. I’ve packed as much practical, real-world knowledge and case-studies that I
could,buttopicsdiscussedaren’texhaustive.Ifyouwantthedocs,youcangrabthemoffGoogle,
because there’s nothing I hate more than buying a book that’s filled with 150 pages of filler
material.
Preface 3
What you need for this book
At a minimum, you need a PHP application that you want to learn how to scale. And you’ll
probably see the most benefit if you have a Linux server that you can test it with. Don’t have
one? I really like Rackspace Cloud³ or Amazon EC2⁴ for testing because you can setup and tear
downserversquickly,andlaunchmulti-servertestsetupscheaply.
What you’ll learn
You’regoingtogetthrownintothedepthsofscalingeverythingfromDNStoMySQLtoNginx.If
yousuddenlywakeuponemorningwith10millionnewusers,don’tevensweatit—thematerial
in this book has been proven to handle it with grace. Today, Twitpic has over 40 million users
and is running the exact setup that I describe in the 10 chapters ahead, it’s the book that Iwish
Ihad4yearsago.
Put what you learn on your Resume. Seriously. If you’re interviewing with startups, there’s
nodoubtthatbeinganexpertinthetopicsdiscussedwillhelpyoulandthejob.Alreadygetthe
job? Or, use some of the techniques to scale your site successfully? Shoot me an email! I really
wanttohearaboutit.
Why you need this book
OkSteve,Igetit,thisbookisfilledwithgreattopicsbutwhatifIdon’tneedtoscaleyet?
System Admin’s are a dying breed, and they’re slowly converging with DevOps⁵. The future
of web programming involves being able to setup your own Nginx server, being able to tune
MySQL,andgetyourproductionserverinorder.
Whenwouldyourathermasterthesetechniques?Whenyou’restillbuilding,stilllearning,and
have the extra time? Or when you’re scrambling around at 2am after you got some media
coverage and have been down for the past 24 hours? Learning to scale is a cheap insurance
policyforsuccess.
Rule Numero Uno
There’s first rule of Scaling PHP Applications is that you will not run the examples in
productionwithouttestingthemfirst.That’sall.Ifyouagree,turnthenextpage.Ifyoudon’t
agree, well, boo, but if you burn down your servers complain to me. Get yourself a cheap test
serveronDigitalOcean⁶orAWS.It’seasy.
³http://www.rackspace.com/cloud/public/servers/
⁴http://aws.amazon.com/ec2/
⁵http://en.wikipedia.org/wiki/DevOps
⁶https://www.digitalocean.com/?refcode=c5806f4bb04f
Ditching the LAMP Stack
What’s wrong with LAMP?
LAMP (Linux, Apache, MySQL, PHP) is the most popular web development stack in the world.
It’s robust, reliable and everyone knows how to use it. So, what’s wrong with LAMP? Nothing.
Youcangoreallyfaronasingleserverwiththedefaultconfigurations.Butwhathappenswhen
you start to really push the envelope? When you have so much traffic or load that your server
isrunningatfullcapacity?
You’ll notice tearing at the seams, and in a pretty consistent fashion too. MySQL is always the
first to go—I/O bound most of the time. Next up, Apache. Loading the entire PHP interpreter
foreachHTTPrequestisn’tcheap,andApache’smemoryfootprintwillproveit.Ifyouhaven’t
crashed yet, Linux itself will start to give up on you—all of those sane defaults that ship with
yourdistributionjustaren’tdesignedforscale.
What can we possibly do to improve on this tried-and-true model? Well, the easiest thing is to
get better hardware (scale vertically)and split the components up (scale horizontally). This will
getyoualittlefurther,butthereisabetterway.Scaleintelligently.Optimize,swappingpiecesof
yourstackforbettersoftware,customizeyourdefaultsandbuildastackthat’sreliableandfault
tolerant.Youwanttospendyourtimebuildinganamazingproduct,notbabysittingservers.
The Scalable Stack
After lots of trial and error, I’ve found what I think is a generic, scalable stack. Let’s call it
LHNMPRR…nothingisgoingtobeascatchyasLAMP!
Linux
WestillhaveOldReliable,butwe’regoingtotunethehelloutofit.Thisbookassumesthelatest
version of Ubuntu Server 12.04, but most recent distributions of Linux should work equally as
well. In some places you may need to substitute apt-get with your own package manager, but
the kernel tweaks and overall concepts should apply cleanly to RHEL, Debian, and CentOS. I’ll
includekernelandsoftwareversionswhereapplicabletohelpavoidanyconfusion.
HAProxy
We’vedumpedApacheandsplititsjobup.HAProxyactsasourloadbalancer—it’sagreatpiece
of software. Many people use nginx as a load balancer, but I’ve found that HAProxy is a better
choiceforthejob.Reasonswhywillbediscussedin-depthinChapter3.
DitchingtheLAMPStack 5
nginx
Nginx is an incredible piece of software⁷. It brings the heat without any bloat and does
webservingextremelywell.InadditiontohavinganincrediblylowmemoryandCPUfootprint,
itisextremelyreliableandcantakeanampleamountofabusewithoutcomplaining.
OneofourbusiestnginxserversatTwitpichandleswellover6,000connectionspersecondwhile
usingonly80MBofmemory.
Anexampleofaserverwithover6000activeconnections
PHP 5.5 / PHP-FPM
ThereareseveralwaystoservePHPapplications:mod_php,cgi,lighttpdfastcgi??.Noneofthese
solutionscomeclosetoPHP-FPM,aFastCGIProcessManagerwrittenbythenginxteamthat’s
beenbundledwithPHPsince5.3.WhatmakesPHP-FPMsoawesome?Well,inadditiontobeing
rock-solid,it’sextremelytunable,providesreal-timestatsandlogsslowrequestssoyoucantrack
andanalyzeslowportionsofyourcodebase.
MySQL
Most folks coming from a LAMP stack are going to be pretty familiar with MySQL, and I will
cover it pretty extensively. For instance, in Chapter 5 you’ll learn how you can get NoSQL
performance out of MySQL. That being said, this book is database agnostic, and most of the
tips can be similarly applied to any database. There are many great databases to choose from:
Postgres, MongoDB, Cassandra, and Riak to name a few. Picking the correct one for your use
caseisoutsidethescopeofthisbook.
⁷Apacheisgreat,too.It’sextremelypopularandwellsupported,butApache+mod_phpisnotthebesttoolforthejobwhenyou’redealing
withhighvolumePHPapps.It’smemoryintensiveanddoesn’tofferfulltuningcapabilities.