Incident Management for Operations

173 Pages·2017·14.078 MB·English

by  Rob Schnepp, Ron Vidal, Chris Hawley

INCIDENT
MANAGEMENT
FOR OPERATIONS
ROB SCHNEPP, RON VIDAL
& CHRIS HAWLEY
Incident Management
for Operations
Rob Schnepp, Ron Vidal, and Chris Hawley
Incident Management for Operations
by Rob Schnepp, Ron Vidal, and Chris Hawley
Copyright © 2017 Rob Schnepp, Ron Vidal, Chris Hawley. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online
editions are also available for most titles (http://oreilly.com/safari). For more information, con-
tact our corporate/institutional sales department: 800-998-9938 or [email protected].
Editor: Brian Anderson Indexer: WordCo Indexing Services, Inc.
Production Editor: Shiny Kalapurakkel Interior Designer: Monica Kamsvaag
Copyeditor: Gillian McGarvey Cover Designer: Karen Montgomery
Proofreader: Kim Cofer Illustrator: Rebecca Demarest
July 2017: First Edition
Revision History for the First Edition
2017-06-20: First Release
See http://www.oreilly.com/catalog/errata.csp?isbn=0636920036159 for release details.
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Incident Management for
Operations, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.
While the publisher and the authors have used good faith efforts to ensure that the information
and instructions contained in this work are accurate, the publisher and the authors disclaim all
responsibility for errors or omissions, including without limitation responsibility for damages
resulting from the use of or reliance on this work. Use of the information and instructions con-
tained in this work is at your own risk. If any code samples or other technology this work con-
tains or describes is subject to open source licenses or the intellectual property rights of others,
it is your responsibility to ensure that your use thereof complies with such licenses and/or
rights.
978-1-491-91762-6
[LSI]
Contents
| Foreword v
| Preface vii
1 | Evaluating the Incident Response PROCESS 1
2 | The Incident Management System (IMS) 17
3 | The Incident Commander (IC) 37
4 | Scaling the Incident Response 91
5 | Unified Command (UC) 103
6 | After Action Review (AAR) 125
| Index 151
iii

Foreword
This book originated from an argument during my first year as Amazon’s “Mas-
ter of Disaster,” as I began applying the incident management and operations
practices I learned as a firefighter to improve Amazon’s overall reliability and
resiliency. I vividly remember facing a room full of scowling engineers and man-
agers who were saying, “I get that these ideas work for firefighters, but do you
really think they can work at internet speed?”
The answer, of course, was yes. The systems and best practices developed
over decades of managing complex emergency incidents—where seconds count
and lives are on the line—work just as well for managing complex incidents for
technology organizations. Over the next few years, my team and I used these
techniques and systems to help transform the culture and technology of what is
now one of the greatest engineering and operations organizations in the world.
When I left Amazon, it was clear to me that as the world was becoming
increasingly connected and distributed, people would come to depend on the
new technology we build and systems we run as part of their daily lives. It was
also clear to me and a group of passionate peers that there were too few people
with the knowledge and experience to build and run these systems at scale. My
friend, Artur Bergman helped me found the O’Reilly Velocity Performance &
Operations conference to organize, develop, and spread our emerging and critical
professional discipline.
As Velocity grew, I started sharing my work with friends and mentors in the
Fire Service. I am fortunate to have worked with and been trained by some of the
most experienced and respected incident management experts in the world, and
I asked them to help build and expand on what I had started.
I convened the first “Web Ops/Fire Ops” summit on a beautiful day at
Artur’s loft in San Francisco. Attending from “the internet” were Artur Bergman
(Fastly/Wikia), John Adams (Twitter), Johnathan Heiliger (Facebook), Pedro
v
vi |
Canahuati (Facebook), Simon Wistow (Fastly), and Chris Brown (Amazon/Chef/
Microsoft). Attending from the “Fire Ops” side were the authors of this book:
Chris Hawley, Rob Schnepp, and Ron Vidal.
After a few hours of sharing backgrounds, “war stories,” and a lot of laugh-
ter, it became clear to everyone that there was both the need and opportunity for
a tech-oriented incident management training program. Chris, Rob, and Ron
formed Blackrock Partners and began consulting with large companies on how to
improve their operations. Since then they have worked with dozens of tech com-
panies, trained thousands of new responders, and reviewed hundreds of inci-
dents as they help companies “work like a fire department at internet speed.”
This book is the first publicly released product of their exceptional work, and
is the essential foundation for building technology and organizations that people
can depend on. I hope you use it.
As we say in the fire department, “See you at the big one!”
—Jesse Robbins
Founder and CEO,
Orion Labs, Inc.
Preface
Does your company have IT incidents?
Have you ever had a high-severity IT incident that disrupted the service your
customers rely on you to provide?
Did that incident damage your company’s reputation, erode trust with your
customers or investors, or create adverse business or financial impact?
Was the incident response and resolution slow, unorganized, or poorly man-
aged?
If you answered “Yes” to any of these questions, ask yourself one more ques-
tion: are you happy with how your company currently responds to IT incidents?
If the answer is “No,” then you must be looking for a better way, and this book
was written for you.
We will show you a new way to think about responding to and resolving IT
incidents. This new way works in small DevOps teams all the way up to the larg-
est enterprise and service provider organizations. Collectively, we have more than
100 years of incident command experience leading special operations teams in
busy urban fire departments on the east and west coasts and managing critical
infrastructure throughout the United States. We also have specialized incident
management experience in more than 40 countries at the highest levels of gov-
ernment and industry. In this book, we identify the best practices from outside
the IT industry that have literally undergone trial by fire, and apply the same inci-
dent response methodology to the world of IT operations.
vii
viii | Preface
For more than 40 years, fire departments throughout the United States
have used a framework to organize and lead the talent and resources
required to respond to emergencies of all kinds. This framework is called
the Incident Management System (IMS) and it’s a simple and effective
way to organize your team rapidly when bad things happen. IMS was
designed as an all-hazard, all-risk incident management framework.
How and why did we make the jump from the fire service to IT? In the fall of
2012, at the suggestion of Jesse Robbins (see Foreword), the original adopter of
using IMS tactics and strategies in IT, we met with a select group of IT professio-
nals who were responsible for uptime at large social media companies and other
large-scale web operations. The goal of the meeting was to discuss and better
understand the challenges faced by IT companies when it comes to resolving IT
incidents. These IT professionals wanted to look outside their industry for a bet-
ter way because the models inside the IT industry were not working well. They
wanted to hear how the fire service manages time-sensitive incidents in which
the stakes are high, the decision-making environment is poor, the conditions are
changing, and the outcomes are uncertain.
As the discussion progressed, the group recognized the striking similarities
between urgent public safety incidents requiring a fire department response and
urgent technology incidents requiring an IT response. In fact, the first IT people
to respond to a service disruption or a cybersecurity attack are the “first respond-
ers,” just like the first firefighters on the scene of a fire. By the end of the meet-
ing, we all agreed that there were significant opportunities to translate the best
parts of the public safety IMS, especially as it pertains to leading people and man-
aging time, directly to the IT industry. The principles are the same and directly
applicable to the type of high severity/priority incidents experienced by the IT
industry.
“A few years ago, we looked for a better way to manage incidents at Sales-
force,” says Kwesi Ames, VP of Site Reliability Engineering at the company. “By
adapting the principles of IMS, the same system used by firefighters, we saw a
tremendous improvement in responding to critical technology incidents. It was a
real game changer for us,” he emphasized. “It provided a disciplined template
that allowed us to be consistently efficient in our response and handling of major
Preface | ix
issues in our production environment. This has resulted in faster recovery times
and minimized loss of customer trust.”
IMS establishes the framework of incident response and the norms of behav-
ior for the incident responders. High severity/priority events place the incident
responders under critical time pressure to resolve IT incidents when customer
trust, adverse financial impacts, and the company’s reputation are at stake. Using
IMS increases your chances at having a good outcome and protecting the compa-
ny’s business.
It is a fact that the future of computing promises more scale, more complex-
ity, and certainly more change—all at greater and greater speed. It’s also true that
the odds increase every day that your organization will have a major technology
incident, created internally or externally.
Without a predictable way to respond to incidents, any organization—grow-
ing or mature—is at risk. Torsten Rueter, global platform services leader at GE
Capital, looks at it this way: “Frameworks such as ITIL have helped mature large-
scale IT operations. The Incident Management System addresses some of the
more subtle—yet extremely important—aspects of managing incidents. Without
strong leadership, collaboration, and shared working patterns, an Incident
Response Team won’t live up to its full potential.”
Building on Jesse Robbin’s work, we adapted the Incident Management Sys-
tem (IMS) from public safety to corporate IT environments. We incorporate IMS
into ITIL, DevOps, Agile, and Lean practices. We collaborate with customers to
build a culture of incident response.
We bring a unique viewpoint to the IT industry. Collectively, we bring over
100 years of fire service and critical infrastructure experience to IT incident man-
agement. We blend our deep global experience in fire, hazardous materials (Haz-
Mat), weapons of mass destruction (WMD), and counterterrorism incident
response with fiber networks, data centers, oil and gas, power, and capital mar-
kets to improve incident management performance.
Our customers generate $400 billion of revenue and create $1 trillion of
market cap, while employing over 850,000 people around the world. These com-
panies rank in the top 10% of the Fortune 500 and PwC Global 100 Software
Leaders, operating globally in the industrial, financial services, consumer prod-
ucts, telecommunications, and software sectors, serving markets in North Amer-
ica, Europe, the Middle East, Africa, Asia, and the Pacific.
Assessment, training, evaluation, and exercises are the best predictor of
future performance. We’ve delivered our incident management programs across