Effective Multi-Tenant Distributed Systems: Challenges and Solutions When Running Complex Environments

102 Pages·2016·34.137 MB·English

by  Chad Carson, Sean Suschter

Effective Multi-Tenant
Distributed Systems
Challenges and Solutions when
Running Complex Environments
Chad Carson and Sean Suchter
BBeeiijjiinngg BBoossttoonn FFaarrnnhhaamm SSeebbaassttooppooll TTookkyyoo
Effective Multi-Tenant Distributed Systems
by Chad Carson and Sean Suchter
Copyright © 2017 Pepperdata, Inc. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA
95472.
O’Reilly books may be purchased for educational, business, or sales promotional use.
Online editions are also available for most titles (http://safaribooksonline.com). For
more information, contact our corporate/institutional sales department:
800-998-9938 or [email protected].
Editor: Nicole Taché and Debbie Hardin Interior Designer: David Futato
Production Editor: Nicholas Adams Cover Designer: Randy Comer
Copyeditor: Octal Publishing Inc. Illustrator: Rebecca Demarest
October 2016: First Edition
Revision History for the First Edition
2016-10-10: First Release
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Effective Multi-
Tenant Distributed Systems, the cover image, and related trade dress are trademarks
of O’Reilly Media, Inc.
While the publisher and the authors have used good faith efforts to ensure that the
information and instructions contained in this work are accurate, the publisher and
the authors disclaim all responsibility for errors or omissions, including without
limitation responsibility for damages resulting from the use of or reliance on this
work. Use of the information and instructions contained in this work is at your own
risk. If any code samples or other technology this work contains or describes is sub‐
ject to open source licenses or the intellectual property rights of others, it is your
responsibility to ensure that your use thereof complies with such licenses and/or
rights.
978-1-491-96183-4
[LSI]
Table of Contents
1. Introduction to Multi-Tenant Distributed Systems. . . . . . . . . . . . . . . . 1
The Benefits of Distributed Systems 1
Performance Problems in Distributed Systems 1
Lack of Visibility Within Multi-Tenant Distributed Systems 4
The Impact on Business from Performance Problems 4
Scope of This Book 6
2. Scheduling in Distributed Systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Introduction 9
Dominant Resource Fairness Scheduling 10
Aggressive Scheduling for Busy Queues 12
Special Scheduling Treatment for Small Jobs 13
Workload-Specific Scheduling Considerations 14
Inefficiencies in Scheduling 16
Summary 22
3. CPU Performance Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Introduction 25
Algorithm Efficiency 26
Kernel Scheduling 27
I/O Waiting and CPU Cache Impacts 30
Summary 32
4. Memory Usage in Distributed Systems. . . . . . . . . . . . . . . . . . . . . . . . . 33
Introduction 33
Physical Versus Virtual Memory 33
Node Thrashing 34
iii
Kernel Out-Of-Memory Killer 37
Implications of Memory-Intensive Workloads for Multi-
Tenant Distributed Systems 38
Summary 41
5. Disk Performance: Identifying and Eliminating Bottlenecks. . . . . . 43
Introduction 43
Overview of Disk Performance Limits 45
Disk Behavior When Using Multiple Disks 46
Disk Performance in Multi-Tenant Distributed Systems 47
Controlling Disk I/O Usage to Improve Performance for
High-Priority Applications 48
Solid-State Drives and Distributed Systems 50
Measuring Performance and Diagnosing Problems 52
Summary 54
6. Network Performance Limits: Causes and Solutions. . . . . . . . . . . . . 55
Introduction 55
Bandwidth Problems in Distributed Systems 55
Other Network-Related Bottlenecks and Problems 61
Measuring Network Performance and Debugging Problems 62
Summary 64
7. Other Bottlenecks in Distributed Systems. . . . . . . . . . . . . . . . . . . . . . 67
Introduction 67
NameNode Contention 67
ResourceManager Contention 69
ZooKeeper 70
Locks 71
External Databases and Related Systems 72
DNS Servers 72
Summary 73
8. Monitoring Performance: Challenges and Solutions. . . . . . . . . . . . . 75
Introduction 75
Why Monitor? 76
What to Monitor 78
Systems and Performance Aspects of Monitoring 80
Algorithmic and Logical Aspects of Monitoring 85
Measuring the Effect of Attempted Improvements 88
Allocating Cluster Costs Across Tenants 89
iv | Table of Contents
Summary 90
9. Conclusion: Performance Challenges and Solutions for Effective
Multi-Tenant Distributed Systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Table of Contents | v

CHAPTER 1
Introduction to Multi-Tenant
Distributed Systems
The Benefits of Distributed Systems
The past few decades have seen an explosion of computing power.
Search engines, social networks, cloud-based storage and comput‐
ing, and similar services now make seemingly infinite amounts of
information and computation available to users across the globe.
The tremendous scale of these services would not be possible
without distributed systems. Distributed systems make it possible for
many hundreds or thousands of relatively inexpensive computers to
communicate with one another and work together, creating the out‐
ward appearance of a single, high-powered computer. The primary
benefit of a distributed system is clear: the ability to massively scale
computing power relatively inexpensively, enabling organizations to
scale up their businesses to a global level in a way that was not possi‐
ble even a decade ago.
Performance Problems in Distributed Systems
As more and more nodes are added to the distributed system and
interact with one another, and as more and more developers write
and run applications on the system, complications arise. Operators
of distributed systems must address an array of challenges that affect
the performance of the system as a whole as well as individual appli‐
cations’ performance.
1