Practical Event-Driven Microservices Architecture

457 Pages·2022·15.976 MB·English

by  Hugo Filipe Oliveira Rocha

Practical
Event-Driven
Microservices
Architecture
Building Sustainable and Highly
Scalable Event-Driven Microservices
—
Hugo Filipe Oliveira Rocha
Practical Event-Driven
Microservices
Architecture
Building Sustainable and Highly
Scalable Event-Driven Microservices
Hugo Filipe Oliveira Rocha
Practical Event-Driven Microservices Architecture: Building Sustainable and
Highly Scalable Event-Driven Microservices
Hugo Filipe Oliveira Rocha
Ermesinde, Portugal
ISBN-13 (pbk): 978-1-4842-7467-5 ISBN-13 (electronic): 978-1-4842-7468-2
https://doi.org/10.1007/978-1-4842-7468-2
Copyright © 2022 by Hugo Filipe Oliveira Rocha
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the
material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now
known or hereafter developed.
Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with
every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an
editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the
trademark.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not
identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to
proprietary rights.
While the advice and information in this book are believed to be true and accurate at the date of publication,
neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or
omissions that may be made. The publisher makes no warranty, express or implied, with respect to the
material contained herein.
Managing Director, Apress Media LLC: Welmoed Spahr
Acquisitions Editor: Smriti Srivastava
Development Editor: Laura Berendson
Coordinating Editor: Shrikant Vishwakarma
Cover designed by eStudioCalamar
Cover image designed by Pexels
Distributed to the book trade worldwide by Springer Science+Business Media LLC, 1 New York Plaza, Suite
4600, New York, NY 10004. Phone 1-800-SPRINGER, fax [phone], e-mail orders-ny@springer-sbm.
com, or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member (owner)
is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware
corporation.
For information on translations, please e-mail [email protected]; for reprint,
paperback, or audio rights, please e-mail [email protected].
Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and
licenses are also available for most titles. For more information, reference our Print and eBook Bulk Sales
web page at http://www.apress.com/bulk-sales.
Any source code or other supplementary material referenced by the author in this book is available to
readers on GitHub via the book’s product page, located at www.apress.com/978- 1- 4842- 7467- 5. For more
detailed information, please visit http://www.apress.com/source- code.
Printed on acid-free paper
As with all the best things in my life, this book included,
started with a suggestion from my wife.
To the love of my life, Eduarda, and my dear daughter, Olívia.
Table of Contents
About the Author ���������������������������������������������������������������������������������������������������xiii
About the Technical Reviewer ���������������������������������������������������������������������������������xv
Acknowledgments �������������������������������������������������������������������������������������������������xvii
Introduction ������������������������������������������������������������������������������������������������������������xix
Chapter 1: Embracing Event-Driven Architectures ���������������������������������������������������1
1.1 The Truth About Monoliths.......................................................................................................4
1.1.1 Anatomy of a Typical Monolith ........................................................................................4
1.1.2 What They Don’t Tell You About Monoliths; It’s Not All Bad .............................................9
1.1.3 When Monoliths Become the Business Constrictor Knot ..............................................13
1.1.4 Using Event-Driven Architectures to Move Away from a Monolith ...............................16
1.2 What Are Microservices and How Do They Relate to Event-Driven .......................................20
1.2.1 D eployment ..................................................................................................................21
1.2.2 Decoupled and Autonomously Developed ....................................................................22
1.2.3 D ata Ownership ............................................................................................................22
1.3 SOA, Microservice, and Event-Driven Architectures ..............................................................22
1.3.1 SOA ...............................................................................................................................23
1.3.2 M icroservice Architecture ............................................................................................24
1.3.3 E vent-Driven Microservice Architecture .......................................................................26
1.4 The Promise of Event- Driven Microservices .........................................................................27
1.4.1 E volving Architecture ....................................................................................................28
1.4.2 Deployment ..................................................................................................................28
1.4.3 Team’s Autonomy..........................................................................................................29
1.4.4 Flexible Technology Stack ............................................................................................30
1.4.5 Resilience and Availability ............................................................................................30
v
Table of ConTenTs
1.4.6 Tunable Scaling ............................................................................................................32
1.4.7 The Past on Demand .....................................................................................................32
1.5 When Should You Use Event-Driven Microservices? .............................................................33
1.6 Overview of the Challenges in Event-Driven Architectures ...................................................36
1.7 Summary...............................................................................................................................38
Chapter 2: Moving from a Monolith to an Event-Driven Architecture���������������������41
2.1 Is Migrating to an Event- Driven Architecture Your Best Option? ...........................................43
2.2 Moving to an Event-Driven Architecture, How to Decide Where to Start ...............................46
2.3 Using an Event-Driven Approach to Move Data from a Monolith ..........................................52
2.4 Using Change Data Capture (CDC) to Move Data from a Monolith ........................................58
2.4.1 Event-Driven and Change Data Capture (CDC), a Real- World Example .......................62
2.5 Migrating Data from a Monolith: Event-Driven As a Source of Truth for Both Systems ........68
2.6 Incremental Migration from a Monolith to an Event- Driven Architecture: Managing
Dependencies ........................................................................................................................70
2.6.1 Managing a Dependency from a New Event-Driven Service to a Legacy Monolith ......70
2.6.2 Managing a Dependency from a Legacy Application to a New Event-Driven Service .....73
2.7 Gradually Moving Traffic to New Microservices ....................................................................76
2.8 Migrating from a Monolith: Two-Way Synchronization and Living with
Two Sources of Truth ..............................................................................................................79
2.9 Summary...............................................................................................................................82
Chapter 3: Defining an Event-Driven Microservice and Its Boundaries ������������������85
3.1 Building Event-Driven Microservices ....................................................................................87
3.1.1 N-Tier Architectures .....................................................................................................88
3.1.2 Clean Architecture ........................................................................................................89
3.1.3 Event-Driven Microservices: Durable vs. Ephemeral Message Brokers and GDPR ......94
3.1.4 Event-Driven Message Types ........................................................................................98
3.1.5 Event-Driven Microservices: When to Use Documents over Events ...........................102
3.1.6 Common Event-Driven Messaging Patterns ...............................................................104
3.1.7 Event-Driven Service Topologies ................................................................................107
3.1.8 Common Event-Driven Pitfalls and Anti-patterns .......................................................111
vi
Table of ConTenTs
3.2 Organizing Event-Driven Microservice Boundaries .............................................................113
3.2.1 Organizational Composition ........................................................................................114
3.2.2 Likelihood of Changes ................................................................................................115
3.2.3 Type of Data ................................................................................................................115
3.3 Brief and Practical Introduction to Domain- Driven Design and Bounded Contexts ............117
3.3.1 How We Can Apply It in Practice .................................................................................118
3.4 Event-Driven Microservices: The Impact of Aggregate Size and Common Pitfalls .............122
3.5 Request-Driven vs. Event-Driven Services .........................................................................124
3.6 Deciding When to Create a New Microservice or Add Functionality to an Existing One .....127
3.7 Summary.............................................................................................................................130
Chapter 4: Structural Patterns and Chaining Processes ��������������������������������������133
4.1 T he Challenges of Transactional Consistency in Distributed Systems ................................135
4.1.1 Why Move from a Monolithic Database in the First Place? ........................................137
4.1.2 The Limitations of Distributed Transactions ...............................................................138
4.1.3 Managing Multi-step Processes with Sagas ..............................................................143
4.2 Event-Driven Orchestration Pattern ....................................................................................146
4.3 Event-Driven Choreography Pattern ...................................................................................150
4.4 Event-Driven Microservices: Orchestration, Choreography, or Both? .................................154
4.5 Data Retrieval in Event- Driven Architectures and Associated Patterns ..............................156
4.5.1 CQS, CQRS, and When to Use Them ............................................................................160
4.5.2 The Different Flavors of CQRS ....................................................................................164
4.5.3 When and How to Use Event Sourcing .......................................................................166
4.5.4 Concerns and When to Use Event Sourcing ................................................................171
4.5.5 Using Command Sourcing and Its Applicability ..........................................................172
4.6 Building Multiple Read Models in Event-Driven Microservice Architectures ......................173
4.7 The Pitfall of Microservice Spaghetti Architectures and How to Avoid It ............................179
4.7.1 Domain Segregation and Clear Boundaries ................................................................180
4.7.2 Context Maps ..............................................................................................................182
4.7.3 Distributed Tracing .....................................................................................................184
4.8 Summary.............................................................................................................................185
vii
Table of ConTenTs
Chapter 5: How to Manage Eventual Consistency ������������������������������������������������187
5.1 T he Impacts of Eventual Consistency and the Need for Alignment with the Business .......190
5.1.2 Liveliness ....................................................................................................................193
5.1.3 The CAP Theorem in the Real World ...........................................................................195
5.2 Using Event Schema in Event-Driven Microservices to Leverage Eventual Consistency ....199
5.3 Applying Microservice Domain Boundaries to Leverage Eventual Consistency ..................204
5.4 Handling Eventual Consistency Delays with Event Versioning ............................................209
5.5 Saving State in Event-Driven Microservices to Avoid Eventual Consistency ......................211
5.5.1 Buffering State As an Alternative to Persistence ........................................................213
5.6 Tackling Eventual Consistency with the End-to-End Argument: A Real-World Use Case ....215
5.7 For Most Use Cases, It’s Not Eventual If Nobody Notices ....................................................218
5.7.1 Event-Driven Autoscaling Use Case with Prometheus and Kafka ..............................220
5.8 Discussing the Tradeoffs of Typical Eventual Consistency Handling Strategies ..................222
5.9 Summary.............................................................................................................................224
Chapter 6: Dealing with Concurrency and Out-of-Order Messages ����������������������227
6.1 Why Is Concurrency Different in a Monolith from an Event-Driven Architecture? ...............229
6.2 Pessimistic vs. Optimistic Concurrency, When and When Not to Use .................................236
6.2.1 Pessimistic vs. Optimistic Approaches .......................................................................237
6.2.2 Solving Concurrency by Implementation and by Design ............................................238
6.3 Using Optimistic Concurrency .............................................................................................240
6.4 Using Pessimistic Concurrency ...........................................................................................244
6.4.1 Distributed Locks in Event-Driven Microservices .......................................................245
6.4.2 Database Transactions As a Concurrency Approach in Distributed Microservices .....250
6.5 Dealing with Out-of-Order Events .......................................................................................251
6.5.1 How Can Events Lose Their Order? ............................................................................252
6.5.2 Solving Out-of-Order Events with Versioning .............................................................254
6.6 Using End-to-End Message Partitioning to Handle Concurrency and
Guarantee Message Ordering .............................................................................................256
6.6.1 Real-World Example of Event-Driven Message Routing Using Kafka .........................258
viii
Table of ConTenTs
6.6.2 The Relevance of Message Routing and Partitioning in Event-Driven
Microservices ..............................................................................................................263
6.6.3 Using End-to-End Partitioning to Handle Concurrency and Ordering .........................266
6.6.4 Limitations of End-to-End Partitioning in Event-Driven Microservices ......................269
6.7 S ummary.............................................................................................................................272
Chapter 7: Achieving Resilience and Event Processing Reliability in
Event-Driven Microservices ���������������������������������������������������������������������������������275
7.1 Common Failures in Microservice Architectures and How They Relate to
Event-Driven Architectures ..................................................................................................277
7.1.1 Cascading Failures and Event-Driven Services ..........................................................279
7.1.2 Load Balancing and Rate Limiters in Event-Driven Services......................................284
7.2 Understanding Message Delivery Semantics......................................................................288
7.2.1 Exactly-Once Delivery Semantics in Kafka .................................................................293
7.3 Avoiding Inconsistencies When Saving State and Publishing Events in Event-Driven
Microservices .......................................................................................................................293
7.3.1 Event Stream As the Only Source of Truth ..................................................................295
7.3.2 Outbox Pattern in Event-Driven Microservices ...........................................................298
7.3.3 Transactions and Compensating Actions to Avoid Inconsistencies in
Event-Driven Microservices ........................................................................................300
7.4 Applying ACID 2.0 As a Resilience Strategy in Event- Driven Microservices .......................302
7.5 Avoiding Message Leak in Event-Driven Microservices ......................................................306
7.5.1 Poison Events .............................................................................................................308
7.6 Applying Common Resilience Patterns in Event- Driven Microservices ..............................308
7.6.1 Retries As a Resilience Approach in Event-Driven Microservices ..............................309
7.6.2 Circuit Breakers in Event-Driven Microservices .........................................................311
7.7 Recovering Data and Repairing State in Event- Driven Microservices ................................314
7.8 Bulkhead Pattern in Event-Driven Microservices ...............................................................316
7.8.1 Priority Queues ...........................................................................................................318
7.9 Conclusion ..........................................................................................................................319
7.10 Summary...........................................................................................................................320
ix
Table of ConTenTs
Chapter 8: Choosing the Correct Event Schema Design in Event-Driven
Microservices �������������������������������������������������������������������������������������������������������323
8.1 Event Storming and Event-Driven Microservices................................................................324
8.1.1 What Are the Limitations of Event Storming? .............................................................329
8.2 Event Schema: Headers and Envelopes ..............................................................................329
8.2.1 Headers vs. Envelopes in Event Schema ....................................................................330
8.2.2 Relevant Contextual Information in Events .................................................................332
8.3 Town Crier Events Pattern ...................................................................................................334
8.4 Bee Events Pattern ..............................................................................................................337
8.5 The Event Schema Goldilocks Principle ..............................................................................340
8.6 Denormalized Event Schema ..............................................................................................343
8.7 Schema Evolution in Event-Driven Microservices ..............................................................345
8.7.1 Backward Compatibility ..............................................................................................346
8.7.2 Forward Compatibility ................................................................................................347
8.7.3 Full Compatibility ........................................................................................................347
8.7.4 No Compatibility .........................................................................................................348
8.7.5 Managing Changes .....................................................................................................348
8.7.6 Event Stream Versioning ............................................................................................351
8.7.7 Using a Downscaler/Upscaler When Evolving Event Schema .....................................352
8.8 S ummary.............................................................................................................................354
Chapter 9: How to Leverage the User Interface in Event- Driven Microservice
Architectures ��������������������������������������������������������������������������������������������������������357
9.1 Using an Aggregating Layer to Build a UI in a Distributed Microservice Architecture ........359
9.2 Backends for Frontends (BFFs) ...........................................................................................362
9.3 UI Decomposition Pattern in Microservice Architectures ....................................................365
9.3.1 UI Application Decomposition Pattern ........................................................................366
9.3.2 UI Page Decomposition Pattern ..................................................................................368
9.3.3 UI Section Decomposition Pattern ..............................................................................369
9.4 The Limitations of API Composition .....................................................................................371
9.5 Task-Based UIs ....................................................................................................................378
x