Infrastructure as Code Managing Servers in the Cloud

362 Pages·2016·10.75 MB·english

by  Kief Morris

I nfrastructure
as Code
MANAGING SERVERS IN THE CLOUD
Kief Morris

Infrastructure as Code
Managing Servers in the Cloud
Kief Morris
BBeeiijjiinngg BBoossttoonn FFaarrnnhhaamm SSeebbaassttooppooll TTookkyyoo
Infrastructure as Code
by Kief Morris
Copyright © 2016 Kief Morris. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are
also available for most titles (http://safaribooksonline.com). For more information, contact our corporate/
institutional sales department: 800-998-9938 or [email protected].
Editor: Brian Anderson Indexer: Judy McConville
Production Editor: Kristen Brown Interior Designer: David Futato
Copyeditor: Amanda Kersey Cover Designer: Karen Montgomery
Proofreader: Jasmine Kwityn Illustrator: Rong Tang and Rebecca Demarest
June 2016: First Edition
Revision History for the First Edition
2016-06-07: First Release
See http://oreilly.com/catalog/errata.csp?isbn=9781491924358 for release details.
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Infrastructure as Code, the cover
image, and related trade dress are trademarks of O’Reilly Media, Inc.
While the publisher and the author have used good faith efforts to ensure that the information and
instructions contained in this work are accurate, the publisher and the author disclaim all responsibility
for errors or omissions, including without limitation responsibility for damages resulting from the use of
or reliance on this work. Use of the information and instructions contained in this work is at your own
risk. If any code samples or other technology this work contains or describes is subject to open source
licenses or the intellectual property rights of others, it is your responsibility to ensure that your use
thereof complies with such licenses and/or rights.
978-1-491-92435-8
[LSI]
Table of Contents
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Part I. Foundations
1. Challenges and Principles. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Why Infrastructure as Code? 3
What Is Infrastructure as Code? 5
Goals of Infrastructure as Code 5
Challenges with Dynamic Infrastructure 6
Server Sprawl 6
Configuration Drift 7
Snowflake Servers 7
Fragile Infrastructure 8
Automation Fear 9
Erosion 10
Principles of Infrastructure as Code 10
Systems Can Be Easily Reproduced 10
Systems Are Disposable 11
Systems Are Consistent 12
Processes Are Repeatable 12
Design Is Always Changing 13
Practices 13
Use Definition Files 13
Self-Documented Systems and Processes 14
Version All the Things 15
Continuously Test Systems and Processes 16
Small Changes Rather Than Batches 16
iii
Keep Services Available Continuously 17
Antifragility: Beyond “Robust” 17
The Secret Ingredient of Antifragile IT Systems 18
Conclusion 18
What’s Next? 19
2. Dynamic Infrastructure Platforms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
What Is a Dynamic Infrastructure Platform? 21
Requirements for a Dynamic Infrastructure Platform 22
Programmable 23
On-Demand 24
Self-Service 25
Infrastructure Resources Provided by the Platform 25
Compute Resources 26
Storage Resources 26
Network Resources 28
Types of Dynamic Infrastructure Platforms 30
Public IaaS Cloud 30
Community IaaS Cloud 30
Private IaaS Cloud 30
Antipattern: Hand-Cranked Cloud 31
Hybrid and Mixed Cloud Options 32
Bare-Metal Clouds 32
Deciding on a Dynamic Infrastructure Platform 34
Public or Private? 34
Cloud Portability 37
Mechanical Sympathy with the Cloud and Virtualization 38
Conclusion 40
3. Infrastructure Definition Tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Choosing Tools for Infrastructure as Code 42
Requirement: Scriptable Interface 42
Requirement: Unattended Mode for Command-Line Tools 42
Requirement: Support for Unattended Execution 43
Requirement: Externalized Configuration 45
Configuration Definition Files 48
Reusability with Configuration Definitions 49
Working with Infrastructure Definition Tools 50
Provisioning Infrastructure with Procedural Scripts 51
Defining Infrastructure Declaratively 52
Using Infrastructure Definition Tools 54
Configuring Servers 54
iv | Table of Contents
Configuration Registries 55
Lightweight Configuration Registries 56
Is a Configuration Registry a CMDB? 57
The CMDB Audit and Fix Antipattern 58
The Infrastructure-as-Code Approach to CMDB 58
Conclusion 59
4. Server Configuration Tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Goals for Automated Server Management 62
Tools for Different Server Management Functions 62
Tools for Creating Servers 63
Tools for Configuring Servers 64
Tools for Packaging Server Templates 65
Tools for Running Commands on Servers 66
Using Configuration from a Central Registry 68
Server Change Management Models 69
Ad Hoc Change Management 69
Configuration Synchronization 69
Immutable Infrastructure 70
Containerized Services 70
Containers 70
Managing Ruby Applications with and without Containers 72
Are Containers Virtual Machines? 73
Using Containers Rather than Virtual Machines 74
Running Containers 75
Security and Containers 76
Conclusion 78
5. General Infrastructure Services. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
Considerations for Infrastructure Services and Tools 81
Prefer Tools with Externalized Configuration 83
Prefer Tools That Assume Infrastructure Is Dynamic 84
Prefer Products with Cloud-Compatible Licensing 84
Prefer Products That Support Loose Coupling 85
Sharing a Service Between Teams 85
Service Instance Templates 86
Monitoring: Alerting, Metrics, and Logging 87
Alerting: Tell Me When Something Is Wrong 87
Metrics: Collect and Analyze Data 89
Log Aggregation and Analysis 89
Service Discovery 90
Server-Side Service Discovery Pattern 91
Table of Contents | v
Client-Side Service Discovery Pattern 91
Distributed Process Management 91
Orchestrating Processes with Server Roles 92
Orchestrating Processes with Containers 92
Scheduling Short Jobs 92
Container Orchestration Tools 92
Software Deployment 93
Deployment Pipeline Software 93
Packaging Software 94
Conclusion 96
Part II. Patterns
6. Patterns for Provisioning Servers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
Server Provisioning 100
A Server’s Life 100
What Goes onto a Server 105
Types of Things on a Server 105
Server Roles 107
Patterns for Creating Servers 108
Antipattern: Handcrafted Server 109
Practice: Wrap Server Creation Options in a Script 110
Antipattern: Hot Cloned Server 111
Pattern: Server Template 111
Antipattern: Snowflake Factory 112
Patterns for Bootstrapping New Servers 112
Pushing to Bootstrap 113
Pulling to Bootstrap 113
Practice: Smoke Test Every New Server Instance 114
Conclusion 115
7. Patterns for Managing Server Templates. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Stock Templates: Can’t Someone Else Do It? 117
Provisioning Servers Using Templates 118
Provisioning at Creation Time 118
Provisioning in the Template 119
Balancing Provisioning Across Template and Creation 120
The Process for Building a Server Template 121
Creating Templates for Multiple Platforms 122
Origin Images 123
Antipattern: Hot Cloned Server Template 123
vi | Table of Contents
Baking a Template from an OS Installation Image 123
Baking a Template from a Stock Image 124
Building a Template from a Unikernel 125
Customizing a Server Template without Booting It 125
Updating Server Templates 126
Reheating a Template 126
Baking a Fresh Template 126
Versioning Server Templates 126
Building Templates for Roles 129
Pattern: Layered Template 129
Sharing Base Scripts for Templates 130
Automating Server Template Management 131
Customizing Servers Before Baking 131
Practice: Automatically Test Server Templates 132
Conclusion 132
8. Patterns for Updating and Changing Servers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Models for Server Change Management 134
Ad Hoc Change Management 134
Continuous Configuration Synchronization 134
Immutable Servers 135
Containerized Servers 136
General Patterns and Practices 136
Practice: Minimize Server Templates 136
Practice: Replace Servers When the Server Template Changes 137
Pattern: Phoenix Servers 137
Patterns and Practices for Continuous Deployment 138
Pattern: Masterless Configuration Management 139
Practice: Apply Cron 139
Continuous Synchronization Flow 140
The Unconfigured Country 141
Patterns and Practices for Immutable Servers 143
Server Image as Artifact 144
Simplifying Confirmation Management Tooling with Immutable Servers 144
Immutable Server Flow 144
Bootstrap Configuration with Immutable Servers 145
Transactional Server Updates 147
Practices for Managing Configuration Definitions 148
Practice: Keep Configuration Definitions Minimal 148
Organizing Definitions 149
Practice: Use Test-Driven Development (TDD) to Drive Good Design 149
Conclusion 150
Table of Contents | vii
9. Patterns for Defining Infrastructure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
Environments 152
Antipattern: Handcrafted Infrastructure 153
Defining Infrastructure Stacks as Code 153
Antipattern: Per-Environment Definition Files 155
Pattern: Reusable Definition Files 155
Practice: Test and Promote Stack Definitions 157
Self-Service Environments 158
Organizing Infrastructure 158
Antipattern: Monolithic Stack 158
Avoid “Lift and Shift” When Migrating Infrastructure 160
Dividing an Application Environment into Multiple Stacks 160
Managing Configuration Parameters Between Stacks 162
Sharing Infrastructure Elements 164
Practice: Manage Application Code and Infrastructure Code Together 166
Approaches to Sharing Definitions 167
Practice: Align Infrastructure Design with the Scope of Change 168
Example: An Infrastructure Design for Microservices 169
Running Definition Tools 174
Conclusion 175
Part III. Practices
10. Software Engineering Practices for Infrastructure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
System Quality 180
Poor-Quality Systems Are Difficult to Change 180
High-Quality Systems Are Easier and Safer to Change 181
Infrastructure Quality Through Code 181
Fast Feedback 181
VCS for Infrastructure Management 182
What to Manage in a VCS 182
Continuous Integration (CI) 183
Continuously Testing Branches Is Not Continuous Integration 183
Who Broke the Build? 185
Ignoring Tests That Fail 186
CI for Infrastructure 187
Continuous Delivery (CD) 187
The Problem with the Integration Phase 187
Deployment Pipelines and Change Pipelines 188
Continuous Delivery Is Not Continuous Deployment 189
Code Quality 190
viii | Table of Contents