Serverless Architectures on AWS. With examples using AWS Lambda

378 Pages·2017·28.21 MB·English

by  Peter Sbarski

Peter Sbarski

F
OREWORDS BY
Patrick Debois
Donald F. Ferguson
M A N N I N G
Serverless Architectures
on AWS
PETER SBARSKI
with Forewords by Patrick Debois
and Donald F. Ferguson
MANNING
SHELTER ISLAND
For online information and ordering of this and other Manning books, please visit
www.manning.com. The publisher offers discounts on this book when ordered in quantity.
For more information, please contact
Special Sales Department
Manning Publications Co.
20 Baldwin Road
PO Box 761
Shelter Island, NY 11964
Email: [e-mail]
©2017 by Manning Publications Co. All rights reserved.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in
any form or by means electronic, mechanical, photocopying, or otherwise, without prior written
permission of the publisher.
Many of the designations used by manufacturers and sellers to distinguish their products are
claimed as trademarks. Where those designations appear in the book, and Manning
Publications was aware of a trademark claim, the designations have been printed in initial caps
or all caps.
Recognizing the importance of preserving what has been written, it is Manning’s policy to have
the books we publish printed on acid-free paper, and we exert our best efforts to that end.
Recognizing also our responsibility to conserve the resources of our planet, Manning books
are printed on paper that is at least 15 percent recycled and processed without the use of
elemental chlorine.
Manning Publications Co. Development editor: Toni Arritola
20 Baldwin Road Technical development editor: Kostas Passadis
PO Box 761 Project editors: Kevin Sullivan and Janet Vail
Shelter Island, NY 11964 Copyeditor: Linda Recktenwald
Proofreader: Melody Dolab
Technical proofreader: David Fombella Pombal
Typesetter: Dottie Marsico
Cover designer: Marija Tudor
ISBN 9781617293825
Printed in the United States of America
1 2 3 4 5 6 7 8 9 10 – EBM – 22 21 20 19 18 17
To my mum and dad,
who always supported and encouraged my passion
for computing

contents
foreword by Patrick Debois xi
foreword by Dr. Donald F. Ferguson xii
preface xv
acknowledgments xvii
about this book xix
about the author xxi
about the cover xxii
PART 1 FIRST STEPS.......................................................1
1 Going serverless 3
1.1 How we got to where we are 4
Service-oriented architecture and microservices 6
Software design 7
1.2 Principles of serverless architectures 9
Use a compute service to execute code on demand 9
Write single-purpose stateless functions 10 ■ Design push-based,
event-driven pipelines 10 ■ Create thicker, more powerful
front ends 10 ■ Embrace third-party services 12
1.3 Transitioning from a server to services 12
1.4 Serverless pros and cons 12
Decision drivers 12 ■ When to use serverless 14
1.5 Summary 15
v
vi CONTENTS
2 Architectures and patterns 16
2.1 Use cases 16
Application back end 17 ■ Data processing and
manipulation 17 ■ Real-time analytics 18 ■ Legacy API
proxy 18 ■ Scheduled services 18 ■ Bots and skills 19
2.2 Architectures 19
Compute as back end 19 ■ Legacy API proxy 24 ■ Hybrid 25
GraphQL 27 ■ Compute as glue 28 ■ Real-time processing 30
2.3 Patterns 31
Command pattern 32 ■ Messaging pattern 33 ■ Priority
queue pattern 34 ■ Fan-out pattern 35 ■ Pipes and filters
pattern 36
2.4 Summary 38
3 Building a serverless application 39
3.1 24-Hour Video 40
General requirements 41 ■ Amazon Web Services 42
Creating your first Lambda function 44 ■ Naming your
Lambda 46 ■ Testing locally 46 ■ Deploying to AWS 48
Connecting S3 to Lambda 50 ■ Testing in AWS 51
Looking at logs 52
3.2 Configuring Simple Notification Service 54
Connecting SNS to S3 54 ■ Getting email from SNS 57
Testing SNS 57
3.3 Setting video permissions 57
Creating the second function 57 ■ Configuring and
securing 58 ■ Testing the second function 59
3.4 Generating metadata 60
Creating the third function and FFprobe 60
3.5 Finishing touches 63
3.6 Exercises 64
3.7 Summary 65
4 Setting up your cloud 66
4.1 Security model and identity management 67
Creating and managing IAM users 67 ■ Creating groups 70
Creating roles 73 ■ Resources 74 ■ Permissions and
policies 74
CONTENTS vii
4.2 Logging and alerting 76
Setting up logging 77 ■ Log retention 78 ■ Filters, metrics,
and alarms 78 ■ Searching log data 80 ■ S3 and logging 80
More on alarms 81 ■ CloudTrail 85
4.3 Costs 87
Creating billing alerts 87 ■ Monitoring and optimizing
costs 88 ■ Using the Simple Monthly Calculator 89
Calculating Lambda and API Gateway costs 90
4.4 Exercises 92
4.5 Summary 93
PART 2 CORE IDEAS ....................................................95
5 Authentication and authorization 97
5.1 Authentication in a serverless environment 98
A serverless approach 98 ■ Amazon Cognito 100 ■ Auth0 101
5.2 Adding authentication to 24-Hour Video 102
The plan 102 ■ Invoking Lambda directly 104
24-Hour Video website 104 ■ Auth0 configuration 106
Adding Auth0 to the website 108 ■ Testing Auth0
integration 113
5.3 Integration with AWS 114
User profile Lambda 115 ■ API Gateway 118
Mappings 121 ■ Invoking Lambda via API
Gateway 124 ■ Custom authorizer 125
5.4 Delegation tokens 129
Real-world examples 130 ■ Provisioning delegation tokens 130
5.5 Exercises 130
5.6 Summary 131
6 Lambda the orchestrator 133
6.1 Inside Lambda 133
Event models and sources 134 ■ Push and pull event
models 135 ■ Concurrent executions 136 ■ Container
reuse 137 ■ Cold and warm Lambda 137
6.2 Programming model 139
Function handler 139 ■ Event object 139 ■ Context
object 140 ■ Callback function 141 ■ Logging 142
viii CONTENTS
6.3 Versioning, aliases, and environment variables 142
Versioning 142 ■ Aliases 144 ■ Environment variables 146
6.4 Using the CLI 148
Invoking commands 148 ■ Creating and deploying
functions 149
6.5 Lambda patterns 151
Async waterfall 152 ■ Series and parallel 157 ■ Using
libraries 158 ■ Move logic to another file 161
6.6 Testing Lambda functions 161
Testing locally 161 ■ Writing tests 162 ■ Testing in
AWS 164
6.7 Exercises 166
6.8 Summary 166
7 API Gateway 168
7.1 API Gateway as the interface 169
Integration with AWS services 170 ■ Caching, throttling, and
logging 170 ■ Staging and versioning 171 ■ Scripting 171
7.2 Working with the API Gateway 171
The plan 173 ■ Creating the resource and method 174
Configuring method execution 177 ■ The Lambda
function 180 ■ Updating the website 184
7.3 Optimizing the gateway 187
Throttling 187 ■ Logging 189 ■ Caching 191
7.4 Stages and versions 194
Creating a stage variable 195 ■ Using stage variables 195
Versions 196
7.5 Exercises 198
7.6 Summary 198
PART 3 GROWING YOUR ARCHITECTURE....................199
8 Storage 201
8.1 Smarter storage 201
Versioning 202 ■ Hosting a static website 204 ■ Storage
classes 206 ■ Object lifecycle management 207 ■ Transfer
acceleration 209 ■ Event notifications 209
CONTENTS ix
8.2 Secure upload 211
Architecture 212 ■ Upload policy Lambda 213 ■ S3 CORS
configuration 218 ■ Uploading from the website 219
8.3 Restricting access to files 223
Removing public access 223 ■ Generating presigned URLs 224
8.4 Exercises 225
8.5 Summary 225
9 Database 227
9.1 Introduction to Firebase 228
Data structure 228 ■ Security rules 230
9.2 Adding Firebase to 24-Hour Video 230
Architecture 231 ■ Setting up Firebase 233 ■ Modifying
Transcode Video Lambda 234 ■ Transcode Video Firebase
Update 239 ■ Connecting Lambda 241 ■ Website 242
End-to-end testing 248
9.3 Securing access to files 248
Signed URL Lambda 249 ■ API Gateway settings 250
Updating the website again 251 ■ Improving
performance 251 ■ Improving Firebase security 254
9.4 Exercises 259
9.5 Summary 259
10 Going the last mile 260
10.1 Deployment and frameworks 260
10.2 Toward better microservices 261
Handling errors 264
10.3 Step Functions 266
Image-processing example 267
10.4 AWS Marketplace 272
10.5 Where from here 274
appendix A Services for your serverless architecture 277
appendix B Installation and setup 282
appendix C More about authentication and authorization 293
appendix D Lambda insider 299
appendix E Models and mapping 305