Python Microservices Development: Build efficient and lightweight microservices using the Python tooling ecosystem, 2nd Edition

311 Pages·2021·4.145 MB·English

by  Simon Fraser, Tarek Ziadé

Python Microservices
Development
Second Edition
Build efficient and lightweight microservices using the
Python tooling ecosystem
Simon Fraser
Tarek Ziadé
BIRMINGHAM—MUMBAI
Python Microservices Development
Second Edition
Copyright © 2021 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system,
or transmitted in any form or by any means, without the prior written permission of the
publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the
information presented. However, the information contained in this book is sold without
warranty, either express or implied. Neither the author(s), nor Packt Publishing or its
dealers and distributors, will be held liable for any damages caused or alleged to have
been caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.
Producer: Shailesh Jain
Acquisition Editor – Peer Reviews: Saby D'Silva
Project Editor: Rianna Rodrigues
Content Development Editor: Alex Patterson
Copy Editor: Safis Editing
Technical Editor: Karan Sonawane
Proofreader: Safis Editing
Indexer: Rekha Nair
Presentation Designer: Pranit Padwal
First published: July 2017
Second edition: September 2021
Production reference: 1280921
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham
B3 2PB, UK.
ISBN 978-1-80107-630-2
www.packt.com
Contributors
About the authors
Simon Fraser is a Site Reliability Engineer for Cisco Meraki. He has over twenty
years of experience in computing, both developing and running systems. He has
worked as a Systems Administrator for an internet service provider; an infrastructure
and high-performance computing engineer at the Welcome Sanger Institute, and as
a Firefox Release Engineer at Mozilla. He studied Cybernetics and Computer Science
at the University of Reading and has also taught academic courses in programming
for scientists.
I would like to thank Emma Apted and Helen Cook for their frequent
support and encouragement, ranging from proofreading to miming a book
opening when I'm clearly not working. My father George Fraser passed on
his immeasurable skill at finishing a project just before the deadline, and
my mother Elizabeth Fraser showed me how to explain concepts and inform
people through years of teaching. This book would also not be possible
without Tarek Ziadé's work on the 1st edition and the solid base on which to
build, as well as the work of Development Editor Alex Patterson, Reviewer
William Kahn-Greene, and Project Editor Rianna Rodrigues.
We might stand on the shoulders of giants, but we also build on a huge
structure made by every one of us, so I would also like to thank everyone
I've worked with over the years in all my different teams. Yes, even you.
Tarek Ziadé is a Software Engineer, located in Burgundy, France. He works at
Elastic, building tools for developers. Before Elastic, he worked at Mozilla for 10 years,
and he founded a French Python user group, called AFPy. Tarek has also written
several articles about Python for various magazines, and a few books in French
and English.
I would like to thank Freya, Suki, Milo, and Amina for being so supportive
of all my book projects.
About the reviewer
Will Kahn-Greene has been building things in Python and other languages
since the late 90s. He manages the crash ingestion pipeline at Mozilla and maintains
several Python libraries. When he's waiting for CI to run tests, he builds things with
wood, tends his tomato plants, and cooks for four.
Table of Contents
Preface xi
Chapter 1: Understanding Microservices 1
The origins of service-oriented architecture 2
The monolithic approach 3
The microservice approach 8
Microservice benefits 10
Separation of concerns 10
Smaller projects 10
Scaling and deployment 11
Pitfalls of microservices 12
Illogical splitting 12
More network interactions 13
Data storing and sharing 13
Compatibility issues 14
Testing 14
Implementing microservices with Python 15
How web services work 16
The WSGI standard 16
Workers, threads, and synchronicity 17
A worker pool approach 17
Being asynchronous 18
Twisted, Tornado, Greenlets, and Gevent 19
Asynchronous Python 19
Language performance 21
Summary 24
[ v ]
Table of Contents
Chapter 2: Discovering Quart 25
Making sure we have Python 27
How Quart handles requests 28
Routing 32
Variables and converters 33
The url_for function 37
Request 37
Response 38
Quart's built-in features 40
The session object 40
Globals 41
Signals 42
Extensions and middleware 44
Templates 46
Configuration 48
Blueprints 50
Error handling 52
Custom error handler 52
A microservice skeleton 54
Summary 57
Chapter 3: Coding, Testing, and Documentation: the Virtuous
Cycle 59
Different kinds of tests 62
Unit tests 63
Functional tests 67
Integration tests 70
Load tests 71
End-to-end tests 75
Using pytest and tox 76
Developer documentation 80
Tools for documentation 81
Version control 85
Continuous Integration and Continuous Deployment 86
GitHub Actions 87
Documentation 88
Coveralls 89
Badges 89
Summary 90
[ vi ]
Table of Contents
Chapter 4: Designing Jeeves 91
The Jeeves bot 92
User stories 92
Monolithic design 93
Model 94
View and template 97
A human-readable view 100
Slack workspaces 103
Taking actions 107
OAuth tokens 108
Authentication and authorization 109
Background tasks 113
Putting together the monolithic design 116
Summary 118
Chapter 5: Splitting the Monolith 119
Identifying potential microservices 119
Code complexity and maintenance 121
Metrics and Monitoring 122
Logging 128
Splitting a Monolith 131
Feature Flags 135
Refactoring Jeeves 137
Workflow 139
Summary 140
Chapter 6: Interacting with Other Services 141
Calling other web resources 142
Finding out where to go 145
Environment variables 145
Service discovery 145
Transferring data 146
HTTP cache headers 147
GZIP compression 149
Protocol Buffers 152
MessagePack 153
Putting it together 155
Asynchronous messages 155
Message queue reliability 156
Basic queues 156
[ vii ]
Table of Contents
Topic exchanges and queues 157
Publish/subscribe 161
Putting it together 162
Testing 162
Using OpenAPI 164
Summary 165
Chapter 7: Securing Your Services 167
The OAuth2 protocol 168
X.509 certificate-based authentication 170
Token-based authentication 173
The JWT standard 174
PyJWT 176
Using a certificate with JWT 177
The TokenDealer microservice 179
The OAuth implementation 180
Using TokenDealer 184
Securing your code 186
Limiting your application scope 187
Untrusted incoming data 189
Redirecting and trusting queries 192
Sanitizing input data 193
Using Bandit linter 194
Dependencies 195
Web application firewall 195
OpenResty: Lua and nginx 196
Rate and concurrency limiting 199
Other OpenResty features 201
Summary 202
Chapter 8: Making a Dashboard 203
Building a ReactJS dashboard 205
The JSX syntax 205
React components 206
Pre-processing JSX 211
ReactJS and Quart 212
Cross-origin resource sharing 213
Authentication and authorization 215
A note about Micro Frontends 216
Getting the Slack token 218
JavaScript authentication 220
Summary 221
[ viii ]
Table of Contents
Chapter 9: Packaging and Running Python 223
The packaging toolchain 224
A few definitions 225
Packaging 226
The setup.py file 226
The requirements.txt file 231
The MANIFEST.in file 234
Versioning 235
Releasing 238
Distributing 239
Running all microservices 242
Process management 243
Summary 245
Chapter 10: Deploying on AWS 247
What is Docker? 248
Introduction to Docker 251
Running Quart in Docker 255
Docker-based deployments 257
Terraform 259
Service discovery 261
Docker Compose 263
Docker Swarm 265
Kubernetes 267
Summary 272
Chapter 11: What's Next? 275
Automation 275
Scaling 276
Content Delivery Networks 277
Multi-cloud deployments 277
Lambda Functions 277
Expanding monitoring 278
Making promises 278
Summary 280
Other Books You May Enjoy 283
Index 287
[ ix ]